# GPU-resident processing

**Performance-first architecture policy revised 2026-09-26; runtime migration planned.**
Effort: GPU-01. Status and owner: [work board](/docs/WORKBOARD).
This policy applies to [AMR-02](/docs/amrex-alignment/DATA_MODEL_ALIGNMENT),
[TRN-01](/docs/amrex-alignment/SINGLE_LEVEL_TRANSPORT), S2/S3 and thermal work.
It replaces the earlier absolute prohibition on CPU processing and computational
readbacks. This document changes no executable behavior or numerical method.

## Execution policy

The goal is to preserve or improve performance while aligning with AMReX's data
model. Keep working simulation fields and repeated numerical processing on the GPU.
Complete reductions, solver coefficients, field-dependent stability calculations
and numerical acceptance checks there so dependent kernels can consume their
results directly. Do not download bulk simulation fields or reduction partials
for CPU numerical completion during stepping or geometry updates. Do not add a
CPU simulation fallback.

Allow host input/setup work and explicit, bounded control/status and layout
metadata transfers when they preserve measured performance. Examples include
batched solve-completion flags, failure status, component counts for allocation,
and compact refinement tags for host clustering. These are deliberate supported
boundaries, not an exemption for arbitrary CPU numerical work. A few bytes can
still impose a costly synchronization; measure frequency and latency as well as
payload. Moving everything on-device is not itself evidence of an improvement.

## Input, processing and output boundaries

| Boundary | Contract |
| --- | --- |
| Input/setup | Host parsing, validation, initial input construction and one-time static preprocessing are allowed. A CPU-built mesh acceleration structure uploaded once is reasonable. Prefer GPU evaluation for expensive or repeated initialization/geometry work; measure setup cost and transfers separately. Do not turn setup into a per-step CPU fallback. |
| Numerical processing | Fields, geometry updates, halos, fluxes, reduction partials/final results and dependent numerical kernels remain resident. Numerical convergence, compatibility and stability checks are evaluated on-device. Host control must not reconstruct coefficients or finish reductions from downloaded values. |
| Host orchestration/control | Host ownership, allocation and launch scheduling may use bounded compact counts, statuses and layout metadata. A device-computed completion flag may stop further batches of launches; a device-selected timestep/status may be mirrored for scheduling/reporting without repeating its numerical selection on the CPU. Record synchronization and validate freshness. |
| Mesh management | GPU tagging and compaction may produce compact coordinates/counts for host clustering, capacity selection and layout construction. Upload derived connectivity/schedules once per layout revision. Keep field transfer, interpolation, remapping and conservation kernels on-device; do not download dense fields/masks to cluster or remap on the CPU. Bound worst-case metadata, cadence and peak memory. |
| Explicit output/inspection | Compute/reduce/sample requested field products on the GPU and copy only requested completed-state products. Host encoding, display, report formatting and scalar-only output analysis are allowed. Output processing cannot become a hidden numerical feedback path. |

Every permitted control/layout transfer must specify its producer, host consumer,
quantity, payload/count bound, cadence, state/layout identity and synchronization
cost. Define and verify those budgets for the workload before accepting an
implementation; no universal byte threshold is asserted here. Dense tagging can
make even compact coordinates expensive and must be included in capacity tests.
Output-off runs still may use documented control/layout channels, but have no
requested field captures. Output-on runs count captures separately.

Independent host reference calculations in verification remain separate oracles,
not production implementations. CPU-backend simulation passes remain prohibited
by the user's verification directive.

## Current gaps and priorities

Source inspection on 2026-09-26 identifies these entry points. This is a starting
inventory, not an exhaustive transfer audit or measured performance result.

| Area | Current behavior | Target and priority |
| --- | --- | --- |
| [GridBlasGpu](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/GridBlasGpu.cs) | Weighted dot/norm paths download per-block partial sums and accumulate on the host. | First priority: complete double reductions on-device and retain results for GPU consumers. |
| [Shared PCG iteration](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/PcgIteration.cs), [pressure contract](/src/PolyCfd.Gpu/Projection/README) | Shared reductions feed CPU alpha/beta calculations, residual/compatibility checks and iteration control. | Device coefficients, numerical checks and dependent updates. Reuse the shared loop/preconditioner seam; compare bounded status polling against fully device-controlled execution. |
| [Diffusion](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/Diffusion/DiffusionGpu.cs), [method](/src/PolyCfd.Gpu/Diffusion/README) | Chebyshev then colored Gauss-Seidel; audit partials, recurrence weights, stall checks and fallback control involve CPU work. | Device-complete audits and recurrence/check state, preserving audited-buffer semantics, nonfinite handling and the shared iteration budget. Host scheduling may consume bounded device status. |
| [CFL](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/Diagnostics/CflGpu.cs), [session](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/Integration/SimulationSession.cs) | Velocity maxima feed host timestep calculations; time/dt are currently host-owned across APIs. | GPU stability calculation and enforcement; migrate authoritative numerical time state and consumers deliberately. A compact scheduling/report mirror is allowed. |
| [Pressure components](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/Projection/PressureComponentsGpu.cs) | Mean subtraction is on-device; more than 16 components triggers classification from three downloaded cell arrays. Other paths read counts/metadata and finish compatibility measurements on the host. | Replace bulk classification with device scans/compaction/reductions; complete compatibility checks on-device. Bounded rebuild counts for host workspace allocation are allowed. |
| [Moving walls](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/Geometry/CutFaceProjectorGpu.cs) | Rigid rotation already has a GPU path; general motion can download face classifications and velocities, modify on CPU and upload. | Preserve rotation performance; implement device-evaluable supported motion instead of bulk-field round trips. |
| [Geometry builder](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/Geometry/CutCellBuilderGpu.cs), [static sources](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Core/Geometry/SdfGeometryDescriptor.cs) | Static host construction, iterative topology counter polling, conditional half-conversion fallback and DEBUG validation captures are distinct paths. | Retain legitimate static setup; migrate repeated numerical work/fallbacks. Budget topology status polling. Record fallback reachability rather than implying every CUDA run uses it; explicit debug captures are output. |
| [Diagnostics](https://github.com/hankbeasley/polycfd/tree/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/Diagnostics), [measurements](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Gpu/Scenarios/ScenarioSectionMeasurements.cs) | Some products already finish GPU reductions; distinguish these from CPU completion of partials or whole-field analysis. | Keep sampling and reductions on-device. Scalar-only presentation/window analysis of requested outputs may run on the host; simulation feedback requires a separately reviewed control contract. |

Historical claims of GPU-resident fields or zero steady-step H2D retain their
original scope. They establish neither closure of this inventory nor performance
of the proposed changes. Static host setup and permitted metadata are no longer
classified as migration gaps solely because a CPU participates.

## Architecture and ordered work

1. **Inventory and freeze evidence.** Trace setup, normal/failed solves, moving
   geometry and output paths. Classify copies and consumers; record bytes, counts,
   frequency, synchronization, allocations and lifetime. Freeze small, representative
   and capacity-limited cases with the same scientific inputs and requested outputs.
2. **Complete reductions and remove bulk round trips.** Introduce resident double
   reduction/scalar views and GPU consumers. Migrate coefficients and numerical
   checks in the shared solver seams, component classification and supported moving
   geometry. Preserve equations, precision, tolerances, iteration caps and failures.
3. **Measure control strategies.** Compare batched completion/status reads with
   device-gated bounded launches. Gating may freeze converged iterates yet still
   enqueue many unused operator/preconditioner kernels. Do not accept extra launch
   or memory cost merely to obtain zero copies. Fully device-controlled loops need
   an ILGPU/CUDA feasibility and performance prototype before production commitment.
4. **Apply the policy to aligned/new algorithms.** Level coordinators launch patch
   kernels and device exchange/flux/coverage operations. Host layout management may
   consume bounded compact metadata. Fully GPU clustering, device allocation and
   adaptive mesh management are later optimizations if measurements justify them.
5. **Audit and qualify each migrated path.** Preserve supported cases and requested
   output, remove replaced CPU numerical paths, and report scientific equivalence,
   transfer budgets and uninstrumented end-to-end performance separately.

Full GPU-01 completion is **not a prerequisite for AMR-02 acceptance**. AMR-02 must
provide appropriate device views/ownership, introduce no new bulk numerical
readbacks and pass unchanged science, memory and performance gates. Record retained
legacy paths against the frozen inventory; unchanged gaps remain GPU-01 work, not
claims of compliance. Avoid combining a structural refactor with an unmeasured
controller rewrite. TRN-01/S2/S3 must satisfy this policy for the behavior they add
or replace; unrelated GPU-01 backlog does not block their acceptance. GPU-01 itself
is complete only when its supported-path migration and audit are accepted.

## Acceptance

- No bulk simulation-field or reduction-partial downloads for CPU numerical work
  in the accepted migrated scope. Reduction results, coefficients and numerical
  checks have device consumers; no CPU fallback is introduced. Trace direct ILGPU
  APIs, wrapper copies, pinned/mapped/managed host access and indirect library
  transfers; report instrumentation blind spots rather than claiming total coverage.
- Every control/status/layout channel has the bounds, cadence, identity and cost
  specified above. Count separately from source uploads and requested output; a
  renamed diagnostic cannot hide numerical feedback. Output-on/off runs both pass.
- Preserve float fields, Half geometry, double weighted reductions, per-component
  pressure gauges, ownership, nonfinite checks and selective capture. Reduction
  reordering requires error/reproducibility evidence; do not lower precision or
  relax scientific checks. GPU placement alone does not improve physical accuracy.
- Qualify supported methods, fixed/adaptive stepping, pressure solvers, regular/cut
  and moving geometry, failures and cancellation. Preserve the supported case set;
  partial coverage cannot establish full GPU-01 completion.
- Apply [architecture performance acceptance](/docs/PERFORMANCE_VALIDATION#architecture-change-acceptance):
  no repeatable regression outside measured noise on each affected representative
  workload, with unchanged references/allowances. Measure setup, steps, rebuilds,
  output, launches, waits, allocations and peak retained memory. Compare on an idle
  machine; a faster large case does not excuse a slower small case. Reject or defer
  changes that fail; ambiguous evidence leaves acceptance pending.
- Runtime implementation requires focused CUDA tests, `py -3.12 tools/verify static fast gpu`
  on Windows (`python3 tools/verify static fast gpu` on Linux), and appropriate
  [verify-all coverage](/docs/COMPLETE_VALIDATION#choose-coverage-for-the-change).
  CPU backend not run (user directive). Documentation-only changes run static checks.

## Why this matches the intended alignment

AMReX combines host management with GPU numerical kernels; its
[GPU model](https://amrex-codes.github.io/amrex/docs_html/GPU.html) does not require
an entirely device-controlled application. In its
[tag collation implementation](https://github.com/AMReX-Codes/amrex/blob/development/Src/AmrCore/AMReX_TagBox.cpp),
`local_collate_gpu` copies counts to the host, constructs offsets there and returns
compact coordinates. These are examples of a boundary, not a performance guarantee
for PolyCFD; pin the compared AMReX revision before implementation.

CUDA provides [conditional graph loops](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#conditional-graph-nodes),
but no production path through our current ILGPU stack has been demonstrated.
Hardware/toolkit compatibility, integration cost and measured performance must be
established before making that execution strategy a dependency.

**Context:** `src/PolyCfd.Gpu`, `src/PolyCfd.Core/Geometry`,
`src/PolyCfd.Execution`, `docs/RUNTIME_CONTRACTS.md`, `docs/PERFORMANCE_VALIDATION.md`.

When migration is accepted, move lasting contracts to the runtime guide and retire
this plan. No equation, reference, tolerance or W4 binding decision changes here.