POLYCFD / Design notebook

AMReX alignment / design proposal / 26 September 2026

From one grid
to a hierarchy.

Give today's solver a clearer structure. Validate better transport on the same grid. Then connect patches and add resolution where the physics needs it.

Four separate efforts. Alignment preserves today's behavior. Conservative transport must prove its single-grid benefit before multiblock and refinement acceptance. These are plans, not delivered capabilities.

Align the model while preserving performance. Keep working fields and repeated numerical processing on the GPU. Allow documented, bounded and measured host control and layout metadata; prohibit bulk field round trips for CPU computation. GPU-01 closes current gaps without blocking unrelated data-model cleanup.

01 / Vocabulary

A domain is a place.
A patch is a piece of storage.

The domain is the physical region being simulated. A level gives it a particular grid spacing. A patch stores a rectangular part of that level.

One coarse patch can cover the entire domain. Fine levels can use several smaller patches. Splitting a level into patches does not change its resolution.

AMReX uses boxes and field collections to express this separation. Official data-model reference.

Physical domainbounds · coordinates · periodicity
Level 0 · spacing h
Patch Aindex box + fields
Patch Bindex box + fields
Proposed structure. During alignment, Level 0 has only one full-domain patch.

Box

Where a patch lives in the level's integer index space.

[lower, upper]

Placement

Where the quantity is sampled relative to a cell.

cell · x/y/z face

Field view

How a kernel reaches values without owning their memory.

bounds + strides + data

Level layout

Which boxes share a resolution and cover which region.

spacing + patch boxes

02 / Follow the same physical domain

The representation changes first.
The mesh changes later.

Choose a stage to inspect the difference. These are schematic 2D grids, not simulation results.

Alignment: one level with one full-domain patch A sixteen by ten cell grid. The alignment effort changes descriptors and ownership while leaving the resolution and physics unchanged. Physical domain · 16 × 10 coarse cells
Level 0 / patch A covers the whole domain. No added resolution.

AMR-02 / same capability

One patch, expressed clearly.

The solver still advances the same grid. Domain coordinates, patch bounds, field placement and memory ownership become explicit and consistently used.

What changes
Data contracts and their consumers.
What must be proved
Existing scientific results, selective capture and performance are preserved.

160 active cells · 1 physical level · 1 patch

Why does keeping one coarse grid not solve all the interface work?

It guarantees base coverage. Fine patches still need values from their neighbors, and the coarse and fine regions must agree on normal volume flux and pressure coupling. The coarse solution must respond to the fine region as part of one physical solution.

Why are allocated and active cells different in the refined illustration?

The fine patch covers 48 of the 160 coarse cells and contains 192 fine cells. We retain 160 + 192 = 352 cell slots, but the physical composite uses 160 − 48 + 192 = 304 active cells. A uniformly fine 2D grid would contain 640 cells. These counts exclude face arrays, ghosts, geometry and solver buffers; they are not a GPU memory or speed prediction. In 3D, a 2:1 refinement divides a coarse cell into eight fine cells.

03 / Improve the foundation first

Balance what moves.
Then measure the accuracy.

Current transport

Trace back and interpolate

Each velocity sample follows the fluid backward and reads an interpolated old value. Our MacCormack correction reduces interpolation error, but does not explicitly balance momentum transferred between neighboring control volumes.

Proposed conservative transport

Use a shared transfer

The same amount leaves one control volume and enters its neighbor. Internal transfers cancel in the total. Boundary flows, pressure, viscosity and applied forces still need their own accounting.

TRN-01 is a numerical effort on the existing grid. Start with current momentum budgets, test a conservative scalar building block, then derive MAC momentum transport and qualify boundaries and cut cells. A scalar test is not proof of momentum conservation or a delivered temperature feature.

Separate questions require separate evidence
QuestionEvidence
Is transport conservative?Momentum budgets close after accounting for physical boundary exchange and sources. Pressure projection's small divergence is a different check.
Is the solution more accurate?Preselected traveling-flow, vortex, separation and cut-cell cases measure field, phase, force and energy errors against appropriate references.
Is the benefit worth its cost?Compare matched grid/timestep results and time to reach a target accuracy. A tighter stability limit can require more steps.

Starting candidate: limited second-order finite-volume transport compatible with our MAC grid; exact method selection remains open. Scope, tests and acceptance. AMReX supports different equation formulations; conservation is our design choice, not a requirement of its data types.

Keep method selection small and explicit.

We already support SemiLagrangian and MacCormack. TRN-01 will retain useful choices and evaluate one conservative candidate. Select one method per run, use it across all patches and levels, and record it with the scientific settings. Each method owns its temporary buffers and declares its timestep, stencil and geometry requirements.

Support is qualified separately for each method and configuration. Initially S3 can require the accepted conservative method. Unsupported combinations must be rejected clearly; the solver must not silently switch methods.

Drill down: method selection and integration · current transport equations · shared-flux derivation.

04 / What already fits

Our MAC fields have a natural home.

Pressure at cell centers, velocity at facesA 2D cell has pressure P in its center, U at the centers of its left and right faces, and V at its top and bottom faces. W is on the out-of-plane faces in three dimensions. P UU VV
2D section of a MAC cell. In 3D, W occupies the Z-normal faces.

Pressure and velocity should not be forced into one identically shaped array. A patch with Nx × Ny × Nz cells has different valid extents for each field.

Pressure
Nx × Ny × Nz
U velocity
(Nx + 1) × Ny × Nz
V velocity
Nx × (Ny + 1) × Nz
W velocity
Nx × Ny × (Nz + 1)

We keep device-resident arrays, float fields, Half geometry and double weighted reductions. Alignment makes their meaning and ownership explicit.

Drill down: physical equations and MAC control volumes · why scalar transport is not enough for momentum.

05 / Where the hard work lives

Three boundaries.
Three different responsibilities.

01

The physical boundary

A wall, inlet, outlet or periodic domain edge. Apply the problem's physical conditions here.

Exists today
02

A same-resolution patch edge

An internal storage split. Obtain neighboring samples and agree on the shared face value; this edge is not a wall.

S2
03

A coarse/fine interface

Different resolutions describe the same fluid. Couple pressure, transfer fields and make weighted fluxes consistent.

S3

S2 must use the accepted transport method's reconstruction, ghost reach and time stages, and apply a consistent flux across shared interfaces. Any retained semi-Lagrangian method also needs patch-aware departure-point sampling in place of current local clamping or wrapping.

Drill down: interface cancellation, average-down and refluxing · cut-cell and moving-boundary balances. References: AMReX boundary filling and AMReX-Hydro projection methods.

06 / Our current code

Adaptable foundations.
Specific assumptions to untangle.

What alignment changes, and what it leaves for later
AreaAMR-02: representationTRN-01 / S2 / S3: behavior
TransportPreserve current equations and numerical choices.TRN-01 validates conservative momentum transport on one grid before extending it to interfaces.
Grid & coordinatesSeparate domain geometry, global boxes and local strides.Support several patches and new refinement layouts.
Session & fieldsOne level with one patch; explicit owners and borrowed views.Coordinate operators and reductions across patches.
Boundary samplingName and guard the current patch-equals-domain restriction.Exchange same-level data, then fill coarse/fine data.
Solid geometrySeparate coordinates from cut-cell metrics; preserve sampling.Establish consistent geometry and fluxes across interfaces.
Results & viewersDerive current metadata from the canonical layout. Preserve wire meanings.Carry native patch/level coverage or label resampled previews.

Physical refinement levels

Where the solution lives

Coarse background plus finer local regions. These levels carry physical simulation state and affect resolution.

Coarse domain → fine region

Multigrid solver levels

How an equation is solved

Auxiliary coarse problems accelerate pressure convergence. They do not add locally refined simulation state.

Pressure problem → coarser corrections

Both can use box descriptors. They need separate identities and ownership. Existing GpuMgHierarchy is the second kind.

Names that make the responsibilities clear

Planned AMR-02 names; these are not current executable APIs
Current namePlanned nameResponsibility
BoundaryPatch / PatchBcSetBoundaryRegion / DomainBoundaryConditionsPhysical boundary regions and conditions, separate from mesh patches.
IGeometryDescriptorIEmbeddedBoundarySourceSolid description used to build numerical embedded-boundary data on the GPU.
CellGeometryEmbeddedBoundaryDataCell and face cut metrics/classifications; the device type follows the same name.
GridBlockMeshPatchA patch within a physical level, after domain/level responsibilities are separated.
GridDescriptorPatchKernelDescriptorA compact derived kernel input; canonical bounds, placement and storage define it.

Separate GridMetrics' coordinate scalars from borrowed embedded-boundary metrics. Share a canonical field-placement description between runtime and output. A level coordinator schedules patch kernels; physical boundary application, same-level exchange and later coarse/fine filling have distinct jobs and explicit stage freshness.

Keep meaningful names such as SimulationSession, MacState, PressureWorkspace and GpuMgHierarchy. Naming, ownership and migration design.

07 / Performance and execution

Keep numerical work resident.
Measure the coordination cost.

Working simulation fields, repeated numerical geometry, transport, solvers and diagnostics belong on the device. Complete reductions and dependent coefficients, convergence evaluation and admissible timestep calculations there. Downloading whole fields or reduction partials to finish those calculations on the CPU is a migration gap, not an acceptable new implementation.

Resident computation

GPU kernels consume resident numerical results. Host code can parse inputs, build one-time static input structures such as a mesh BVH, allocate storage and schedule launches. Explicit, bounded control/status and layout metadata may cross to the host; record the payload, purpose, cadence and measured cost.

Explicit final output

Compute a requested measurement, slice or export product on the GPU first. Its final product can cross to file or presentation code and be counted separately. This output path does not authorize downloading fields for CPU numerical processing that feeds the simulation.

Why allow a small control channel?

The GPU can evaluate convergence and publish a completion flag between batches of iterations so the host stops launching work. Forcing launches through the full iteration cap may waste time even when device flags make the remaining kernels do no numerical work. Small transfers still synchronize and are not automatically cheap: compare their measured cost with an entirely device-controlled loop before choosing. That loop requires an ILGPU/CUDA feasibility and performance prototype, not a blanket prerequisite.

Does AMReX alignment require device-only regridding?

No. Keep tagging and numerical field remapping on the GPU, while allowing bounded compact tag/layout metadata for host clustering and allocation. AMReX itself uses host coordination in GPU tag collation. Fully device-driven clustering and regridding are later optimizations to justify with measurements.

Gate: new or migrated numerical paths have no bulk field downloads or CPU completion of reductions. Audit direct and indirect copies, metadata transfer counts/bytes, synchronization time, launch overhead, peak memory and end-to-end performance. Count requested output separately. Scientific checks, precision and existing performance limits remain binding.

GPU-01 scope, source inventory and acceptance · Algorithm execution details.

08 / Numerical review

Follow an idea all the way
to its equations and evidence.

The algorithm companion starts with the physical model and opens into discrete equations, source code and validation protocols. Current behavior, proposed algorithms and unresolved derivations are labeled separately.

  1. Start with physics and field locationsIncompressible flow, units and staggered control volumes.
  2. Trace today's complete time stepForces, diffusion, pressure signs and continuity.
  3. Inspect today's transport equationsBacktracing, interpolation, correction and limiting.
  4. Derive a conservative updateShared transfers and an illustrative scalar algorithm.
  5. Identify the work still to deriveStaggered momentum, cut volumes and moving walls.
  6. Review the proof and test obligationsConservation, stability, accuracy and physical validation.

09 / Delivery gates

Prove each step before adding the next.

Each effort follows the execution policy for the work it changes. AMR-02 does not require all of GPU-01 to finish; TRN-01, S2 and S3 are not blocked by unrelated migration gaps. New numerical paths must keep their work resident, document bounded host coordination and preserve the applicable scientific, memory and performance gates.

  1. AMR-02

    Align today's structures

    Inventory consumers, introduce canonical descriptors, migrate ownership and callers, then remove superseded representations.

    Gate: existing inputs, fields, measurements, outputs and scientific acceptance remain valid. No new mesh capability, bulk field transfers or abstraction-driven performance/memory regression. Existing GPU-01 gaps remain explicitly tracked.

  2. TRN-01

    Validate selectable single-grid transport

    Measure current budgets, test shared scalar fluxes, formulate MAC momentum transport, then qualify boundaries, cut cells and separately supported motion. Integrate one recorded method choice per run with explicit support, timestep and memory requirements.

    Gate: conservation and stability checks pass, retained methods preserve their scientific gates, and preselected cases demonstrate candidate accuracy benefit with acceptable cost. Selection is reproducible and unsupported configurations are rejected.

  3. S2

    Connect patches at one resolution

    Begin with the same problem split in two. Extend the accepted transport method with shared fluxes and same-level sampling, plus coupled pressure/diffusion, geometry, measurements and output.

    Gate: partitioned and monolithic solutions and momentum budgets agree within declared tolerances, including periodic and cut interfaces. A simple two-patch test alone does not finish S2.

  4. S3 / fixed

    Add resolution in one region

    Start with one coarse base, one aligned 2:1 fine patch and synchronized timesteps. Add coverage masks, transfers and composite equations.

    Gate: accuracy and conservation pass against uniform-fine references, with less measured memory or time at comparable measurement accuracy.

  5. S3 / adaptive

    Let refined regions change

    Extend fixed refinement to multiple patches/levels. Then add GPU tagging and safe field transfer, with bounded compact metadata for host layout management and allocation, plus geometry/solver rebuilding.

    Gate: refine/derefine cycles preserve the required budgets and constraints; metadata transfers and regrid costs are measured. Thermal and moving-geometry combinations need their own evidence. Fully device-driven regridding and time subcycling are later decisions.

S2 layout work can overlap transport development. Its transport interfaces follow TRN-01's established method. Coordinate scalar work with T1, while keeping thermal acceptance separate.

Continue reading

The design behind the diagrams.

The Markdown plans define scope and acceptance. The work board owns current status. Proposed type names and diagrams are explanatory; they are not executable API or file-format specifications.