# GPU-resident processing

**Hard architecture requirement, agreed 2026-09-26; implementation planned.**
Effort: GPU-01. Status and owner: [work board](/docs/WORKBOARD).
This requirement applies to all future simulation work, including
[AMR-02](/docs/amrex-alignment/DATA_MODEL_ALIGNMENT),
[TRN-01](/docs/amrex-alignment/SINGLE_LEVEL_TRANSPORT), S2/S3 and thermal work.
Current code is not fully compliant. This document changes no executable behavior.

## Hard requirement

All simulation processing must execute on the GPU. Device-to-host (D2H) copies
must not be used to perform any part of that processing on the CPU. Keeping only
large fields resident is insufficient: scalar and partial-result readbacks that
feed numerical calculations or decisions are also prohibited. There is no automatic
CPU fallback and no acceptance exemption for an inexpensive or infrequent round trip.

This includes initialization from scientific inputs, geometry discretization and
rebuilding, source evaluation, transport, diffusion, projection, connected-component
and nullspace work, reductions through their final result, solver coefficients and
convergence decisions, timestep selection and enforcement, diagnostics/measurements,
sampling, interface exchange and future refinement/adaptation. Derived numerical
data stays resident for every later consumer.

CPU responsibilities are input/file parsing, configuration and metadata validation,
kernel compilation, resource ownership, launch orchestration and external I/O.
Those roles do not authorize CPU numerical work on downloaded data. Static dispatch
of a method selected in the input is host orchestration; choosing a numerical update
from a downloaded residual, velocity maximum, topology label or tag is processing.

## Input, processing and output boundaries

| Boundary | Contract |
| --- | --- |
| Input/setup | Parse scenario and asset bytes on the host and upload required source data. Evaluate initial fields, numerical geometry/acceleration structures and derived coefficients on the GPU. Do not download results for CPU completion, including during setup. |
| Simulation processing | Keep fields, geometry, halos, partial/final reductions, component information, fluxes, timestep and iterative control state on the device. Device operations consume their predecessors' results directly. Computational D2H calls and bytes must both be zero. |
| Host orchestration | Allocate from declared capacities and schedule kernels/streams using configuration metadata. GPU-resident flags govern numerical convergence, rejection and dependent updates. Observing a completed operation's final status must not replace device-side numerical decisions. |
| Explicit output/inspection | A requested completed-frame export, measurement, status or inspection capture may cross the device boundary for external consumption. Compute/reduce/sample the requested product on the GPU first; copy only that final requested product. File encoding, publication and presentation do not feed numerical results back into the solver. |

The output boundary preserves requested save/view/inspect workflows. It is not a
general permission to download fields for analysis in production. A final scalar
for display is output; a scalar used by the host to form a PCG coefficient or pick
the next timestep is a forbidden processing readback. Report the actual accepted
timestep from the GPU; do not make its host copy authoritative for subsequent kernels.
Output-off runs have no field readback. Any remaining final completion/failure record
must be separately identified and bounded; output payloads cannot hide feedback loops.

Independent reference calculations in verification remain separate oracles. Requested
test captures may be compared against them, but neither host reference code nor a
validation observer may become the production implementation or a CPU fallback.
CPU-backend simulation passes remain prohibited by the user's verification directive.

## Current gaps to close

Source inspection on 2026-09-26 identifies the following entry points. This is a
starting inventory, not an exhaustive audit or a measured transfer result.

| Area | Current behavior to audit/migrate | Required result |
| --- | --- | --- |
| [GridBlasGpu](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/GridBlasGpu.cs) | Weighted dot/norm paths read per-block partial sums and accumulate them on the host. Some reductions return host scalars. | Complete weighted reductions on-device with double accumulation; retain results for GPU consumers. |
| [Shared PCG iteration](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/PcgIteration.cs), [Jacobi solver](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/PcgSolverGpu.cs), [MG solver](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/Multigrid/PcgMgSolverGpu.cs) | Both solver shells use `PcgIteration`; weighted reductions return host values for recurrence coefficients, residual checks, regional-imbalance refusal and iteration control. | Migrate the shared loop once: device-resident coefficients, convergence/failure flags and dependent updates, preserving the preconditioner seam, equations, tolerances and iteration caps. |
| [Diffusion](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/Diffusion/DiffusionGpu.cs), [implemented method](/src/PolyCfd.Gpu/Diffusion/README) | Chebyshev semi-iteration precedes the colored Gauss-Seidel fallback. Audit partials are copied to the host for final sums/norms; recurrence weights, stall/exit decisions and fallback control are host calculations. | Device-complete audits, recurrence coefficients and numerical decisions for both phases, preserving the operator, tolerance and shared budget. |
| [CflGpu](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/Diagnostics/CflGpu.cs) | Velocity maxima are returned to the host; ratios and the safe timestep are computed there. | Select/enforce timestep and final-time constraints on-device; kernels consume resident time state. |
| [PressureComponentsGpu](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/Projection/PressureComponentsGpu.cs), [implemented region contract](/src/PolyCfd.Gpu/Projection/README#pressure-regions) | Per-region mean subtraction now completes on-device. Rebuild still reads the component count, slot metadata and volumes for host decisions; more than 16 regions selects host classification from downloaded labels/anchors/weights. `ProjectAndMeasure` reads region sums for host imbalance calculation. | Preserve device mean subtraction; migrate all classification, workspace selection, compatibility measurement and refusal decisions to resident processing for every supported topology. |
| [CutCellBuilderGpu](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/Geometry/CutCellBuilderGpu.cs), [geometry sources](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Core/Geometry/IGeometryDescriptor.cs) | Host conversion fallback and topology-change readback paths exist; some initialization/build routes originate in CPU geometry. Debug validation also downloads arrays. Reachability must be recorded per supported case. | GPU setup/rebuild/conversion/topology and initialization. Explicit final debug captures remain output; CPU processing fallback is rejected or removed, never silently selected. |
| [Diagnostics](https://github.com/hankbeasley/polycfd/tree/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/Diagnostics), [scenario measurements](https://github.com/hankbeasley/polycfd/blob/18102808d7066ed97d8908d66755a68df8d0516e/src/PolyCfd.Gpu/Scenarios/ScenarioSectionMeasurements.cs), capture/export callers | Distinguish completed requested products from downloads whose numerical work is finished on the host. | GPU sampling, reductions, windows and derived quantities, followed only by requested final-product publication. |

Current docs and historical receipts saying "GPU-resident fields" or "zero steady-step
H2D" do not establish this stronger D2H requirement. Preserve those results with their
original scope. Existing CPU-backend availability is a current implementation fact,
not an accepted backend or a preservation requirement for new production work.

## Architecture and ordered work

1. **Inventory and freeze evidence.** Trace all processing and copy paths, including
   initialization, normal/failed solves, geometry changes and diagnostics. Classify
   each transfer by purpose, bytes, count, consumer, lifetime and whether the result
   influences future computation. Freeze representative scientific/performance cases.
2. **Establish resident scalar/control storage and full reductions.** Give dependent
   kernels device views of double reduction results, coefficients, time and flags.
   Derive a bounded execution strategy supported by ILGPU/CUDA: ordered launches with
   device gating or another validated mechanism. Do not require a speculative persistent
   kernel or device-side allocation framework. Host loops over fixed metadata are okay;
   host polling of partial numerical results to control iterations is not.
3. **Migrate current solver and geometry paths.** Complete reductions, coefficients,
   convergence, CFL, component means, initialization, geometry and diagnostics on-device.
   Preserve reference behavior and eliminate obsolete production CPU paths. Geometry
   source parsing and kernel-facing device data are distinct responsibilities. Reuse the
   shared `PcgIteration` and existing preconditioner/diffusion seams; migrate their host
   calculations without recreating separate PCG loops or changing the selected methods.
4. **Apply the contract to aligned levels and new algorithms.** Use level coordinators
   to schedule patch kernels, never independent CPU patch solves. Device exchange,
   transfers, flux accumulation and coverage reductions operate on the aligned data.
   For S3 adaptation, compute tags, compaction/clustering, layout-derived numerical
   connectivity and state remapping on the GPU. Design bounded capacities/workspaces;
   report an explicit capacity failure rather than download tags for CPU clustering.
5. **Audit output and prove acceptance.** GPU-compute final requested products and
   separate their publication from processing. Remove CPU completion disguised as
   diagnostics. Record copy and scientific evidence, supported-case coverage and cost.

GPU-01 is a prerequisite to strict runtime acceptance, not a new prerequisite to
writing descriptors or reviewing equations. Its migration can be implemented alongside
AMR-02 where ownership changes overlap. AMR-02 still adds no physics or mesh capability;
new transport equations remain TRN-01. Each supported production path must meet this
requirement; a passed subset does not establish global GPU-01 completion. Unchanged
CPU-assisted subpaths are gaps, not grandfathered exceptions.

## Acceptance

- **Zero computational D2H calls and zero computational D2H bytes**, across setup,
  stepping, geometry rebuilding, solver retries/failures and enabled measurements.
  Include direct ILGPU `GetAsArray`/host-returning reductions, wrapper copies, pinned
  staging, mapped/managed-memory host access and indirect library transfers. A counter
  that sees only `IlgpuBackend.CopyToHost` is not enough. Use code-path review plus
  transfer instrumentation/profiling; report what the instrumentation cannot observe.
- Measure output-off and explicitly requested output-on runs separately. Every output
  readback must identify its request, quantity, completed-state identity and bounded
  payload. No host processing result is re-uploaded or reused to advance the simulation.
  Enabling diagnostics must not silently select a CPU compute path.
- Preserve float fields, Half geometry, double weighted reductions, per-component
  pressure gauges, ownership and selective capture. Reduction reordering is a numerical
  change requiring error/reproducibility evidence, not permission to lower precision
  or relax any scientific limit. Keep nonfinite, compatibility and convergence checks.
- Qualify all supported methods, fixed/adaptive stepping, pressure solvers, regular/cut
  geometry, moving geometry and error/cancellation paths. Missing GPU implementations
  produce explicit unsupported-case errors during development; preserving the existing
  supported case set is required before declaring the migration complete.
- Preserve reference cases and performance allowances. Measure allocations, retained
  memory, launches/barriers and total cost including requested output on an idle GPU.
  Excess inactive iterations or large speculative buffers are not automatically accepted
  merely because readback is zero. Follow [performance acceptance](/docs/PERFORMANCE_VALIDATION#architecture-change-acceptance).
- Runtime implementation requires `py -3.12 tools/verify static fast gpu` on Windows
  (`python3 tools/verify static fast gpu` on Linux), focused CUDA
  tests and appropriate `verify-all` coverage, including full coverage before broad
  numerical acceptance. See [suite selection](/docs/COMPLETE_VALIDATION#choose-coverage-for-the-change).
  CPU backend not run (user directive). This documentation task runs static checks only.

**Context:** `src/PolyCfd.Gpu`, `src/PolyCfd.Core/Geometry`,
`src/PolyCfd.Execution`, `docs/RUNTIME_CONTRACTS.md`, `docs/PERFORMANCE_VALIDATION.md`.

When implementation meets these gates, move the lasting execution/transfer contract
to the runtime guide and retire this plan. No equation, reference, tolerance or W4
binding decision is changed by declaring this requirement.