Box
Where a patch lives in integer index space, with inclusive endpoints.
[SmallEnd, BigEnd]AMReX alignment / design proposal / 26 September 2026
From one grid
to a hierarchy.
Give today's solver a clearer structure. Validate better transport on the same grid. Then connect patches and add resolution where the physics needs it.
Four separate efforts. Alignment preserves today's behavior. Conservative transport must prove its single-grid benefit before multiblock and refinement acceptance. These are plans, not delivered capabilities.
Align the model while preserving performance. Keep working fields and repeated numerical processing on the GPU. Allow documented, bounded and measured host control and layout metadata; prohibit bulk field round trips for CPU computation. GPU-01 closes current gaps without blocking unrelated data-model cleanup.
01 / Vocabulary
A domain is a place.
A patch is a piece of storage.
The domain is the physical region being simulated. A level gives it a particular grid spacing. A patch stores a rectangular part of that level.
One coarse patch can cover the entire domain. Fine levels can use several smaller patches. Splitting a level into patches does not change its resolution.
AMReX uses boxes and field collections to express this separation. Official data-model reference.
Physical domainbounds · coordinates · periodicity
Level 0 · spacing h
Patch Aindex box + fields
Patch Bindex box + fields
IndexType
Cell or node placement per axis; a face is nodal along its normal.
cell · x/y/z faceField view
How a kernel reaches values without owning their memory.
bounds + strides + dataLevel layout
Which boxes share a resolution and cover which region.
spacing + patch boxesAdopt AMReX names and bounds where practical
Approved design direction; implementation remains planned. Use Box, IntVect and IndexType for the matching concepts. Follow AMReX's meaning with C# spelling, such as SmallEnd and BigEnd. Keep a different name where our responsibilities differ, and document why.
An eight-cell axis numbered 0 through 7 has inclusive bounds [0, 7]. A loop using i < end instead needs end = 8. Both visit the same cells. The canonical box stores inclusive bounds; an explicitly derived EndExclusive can serve existing loops without maintaining two independent representations.
Implementation review checks off-by-one errors, one-cell and negative-origin boxes, face extents, empty regions and overflow. It also verifies measured performance. The inclusive convention is decided; reviewing existing half-open loops is a migration check.
Names such as DomainGeometry distinguish coordinate mapping from solid geometry; LevelLayout includes ownership/identity beyond a box collection. Matching terminology does not require new buffers, unused operations or the full AMReX API.
Naming policy and justified differences · Bounds, conversion and acceptance details.
02 / Follow the same physical domain
The representation changes first.
The mesh changes later.
Choose a stage to inspect the difference. These are schematic 2D grids, not simulation results.
AMR-02 / same capability
One patch, expressed clearly.
The solver still advances the same grid. Domain coordinates, patch bounds, field placement and memory ownership become explicit and consistently used.
- What changes
- Data contracts and their consumers.
- What must be proved
- Existing scientific results, selective capture and performance are preserved.
160 active cells · 1 physical level · 1 patch
Why does keeping one coarse grid not solve all the interface work?
It guarantees base coverage. Fine patches still need values from their neighbors, and the coarse and fine regions must agree on normal volume flux and pressure coupling. The coarse solution must respond to the fine region as part of one physical solution.
Why are allocated and active cells different in the refined illustration?
The fine patch covers 48 of the 160 coarse cells and contains 192 fine cells. We retain 160 + 192 = 352 cell slots, but the physical composite uses 160 − 48 + 192 = 304 active cells. A uniformly fine 2D grid would contain 640 cells. These counts exclude face arrays, ghosts, geometry and solver buffers; they are not a GPU memory or speed prediction. In 3D, a 2:1 refinement divides a coarse cell into eight fine cells.
03 / Improve the foundation first
Balance what moves.
Then measure the accuracy.
Current transport
Trace back and interpolate
Each velocity sample follows the fluid backward and reads an interpolated old value. Our MacCormack correction reduces interpolation error, but does not explicitly balance momentum transferred between neighboring control volumes.
Proposed conservative transport
Use a shared transfer
The same amount leaves one control volume and enters its neighbor. Internal transfers cancel in the total. Boundary flows, pressure, viscosity and applied forces still need their own accounting.
TRN-01 is a numerical effort on the existing grid. Start with current momentum budgets, test a conservative scalar building block, then derive MAC momentum transport and qualify boundaries and cut cells. A scalar test is not proof of momentum conservation or a delivered temperature feature.
| Question | Evidence |
|---|---|
| Is transport conservative? | Momentum budgets close after accounting for physical boundary exchange and sources. Pressure projection's small divergence is a different check. |
| Is the solution more accurate? | Preselected traveling-flow, vortex, separation and cut-cell cases measure field, phase, force and energy errors against appropriate references. |
| Is the benefit worth its cost? | Compare matched grid/timestep results and time to reach a target accuracy. A tighter stability limit can require more steps. |
Starting candidate: limited second-order finite-volume transport compatible with our MAC grid; exact method selection remains open. Scope, tests and acceptance. AMReX supports different equation formulations; conservation is our design choice, not a requirement of its data types.
Keep method selection small and explicit.
We already support SemiLagrangian and MacCormack. TRN-01 will retain useful choices and evaluate one conservative candidate. Select one method per run, use it across all patches and levels, and record it with the scientific settings. Each method owns its temporary buffers and declares its timestep, stencil and geometry requirements.
Support is qualified separately for each method and configuration. Initially S3 can require the accepted conservative method. Unsupported combinations must be rejected clearly; the solver must not silently switch methods.
Drill down: method selection and integration · current transport equations · shared-flux derivation.
04 / What already fits
Our MAC fields have a natural home.
Pressure and velocity should not be forced into one identically shaped array. A patch with Nx × Ny × Nz cells has different valid extents for each field.
- Pressure
- Nx × Ny × Nz
- U velocity
- (Nx + 1) × Ny × Nz
- V velocity
- Nx × (Ny + 1) × Nz
- W velocity
- Nx × Ny × (Nz + 1)
We keep device-resident arrays, float fields, Half geometry and double weighted reductions. Alignment makes their meaning and ownership explicit.
Drill down: physical equations and MAC control volumes · why scalar transport is not enough for momentum.
05 / Where the hard work lives
Three boundaries.
Three different responsibilities.
The physical boundary
A wall, inlet, outlet or periodic domain edge. Apply the problem's physical conditions here.
A same-resolution patch edge
An internal storage split. Obtain neighboring samples and agree on the shared face value; this edge is not a wall.
A coarse/fine interface
Different resolutions describe the same fluid. Couple pressure, transfer fields and make weighted fluxes consistent.
S2 must use the accepted transport method's reconstruction, ghost reach and time stages, and apply a consistent flux across shared interfaces. Any retained semi-Lagrangian method also needs patch-aware departure-point sampling in place of current local clamping or wrapping.
Drill down: interface cancellation, average-down and refluxing · cut-cell and moving-boundary balances. References: AMReX boundary filling and AMReX-Hydro projection methods.
06 / Our current code
Adaptable foundations.
Specific assumptions to untangle.
| Area | AMR-02: representation | TRN-01 / S2 / S3: behavior |
|---|---|---|
| Transport | Preserve current equations and numerical choices. | TRN-01 validates conservative momentum transport on one grid before extending it to interfaces. |
| Grid & coordinates | Separate domain geometry, global boxes and local strides. | Support several patches and new refinement layouts. |
| Session & fields | One level with one patch; explicit owners and borrowed views. | Coordinate operators and reductions across patches. |
| Boundary sampling | Name and guard the current patch-equals-domain restriction. | Exchange same-level data, then fill coarse/fine data. |
| Solid geometry | Separate coordinates from cut-cell metrics; preserve sampling. | Establish consistent geometry and fluxes across interfaces. |
| Results & viewers | Derive current metadata from the canonical layout. Preserve wire meanings. | Carry native patch/level coverage or label resampled previews. |
Physical refinement levels
Where the solution lives
Coarse background plus finer local regions. These levels carry physical simulation state and affect resolution.
Coarse domain → fine region
Multigrid solver levels
How an equation is solved
Auxiliary coarse problems accelerate pressure convergence. They do not add locally refined simulation state.
Pressure problem → coarser corrections
Both can use box descriptors. They need separate identities and ownership. Existing GpuMgHierarchy is the second kind.
Prefer AMReX names where the responsibilities match
Adopt Box, IntVect and IndexType with inclusive bounds. Keep the application-specific names below where they clarify different responsibilities; review the convention and practical differences.
| Current name | Planned name | Responsibility |
|---|---|---|
BoundaryPatch / PatchBcSet | BoundaryRegion / DomainBoundaryConditions | Physical boundary regions and conditions, separate from mesh patches. |
IGeometryDescriptor | IEmbeddedBoundarySource | Solid description used to build numerical embedded-boundary data on the GPU. |
CellGeometry | EmbeddedBoundaryData | Cell and face cut metrics/classifications; the device type follows the same name. |
GridBlock | MeshPatch | A patch within a physical level, after domain/level responsibilities are separated. |
GridDescriptor | PatchKernelDescriptor | A compact derived kernel input; canonical bounds, placement and storage define it. |
Separate GridMetrics' coordinate scalars from borrowed embedded-boundary metrics. Share a canonical field-placement description between runtime and output. A level coordinator schedules patch kernels; physical boundary application, same-level exchange and later coarse/fine filling have distinct jobs and explicit stage freshness.
Keep meaningful names such as SimulationSession, MacState, PressureWorkspace and GpuMgHierarchy. Naming, ownership and migration design.
07 / Performance and execution
Keep numerical work resident.
Measure the coordination cost.
Working simulation fields, repeated numerical geometry, transport, solvers and diagnostics belong on the device. Complete reductions and dependent coefficients, convergence evaluation and admissible timestep calculations there. Downloading whole fields or reduction partials to finish those calculations on the CPU is a migration gap, not an acceptable new implementation.
Resident computation
GPU kernels consume resident numerical results. Host code can parse inputs, build one-time static input structures such as a mesh BVH, allocate storage and schedule launches. Explicit, bounded control/status and layout metadata may cross to the host; record the payload, purpose, cadence and measured cost.
Explicit final output
Compute a requested measurement, slice or export product on the GPU first. Its final product can cross to file or presentation code and be counted separately. This output path does not authorize downloading fields for CPU numerical processing that feeds the simulation.
Why allow a small control channel?
The GPU can evaluate convergence and publish a completion flag between batches of iterations so the host stops launching work. Forcing launches through the full iteration cap may waste time even when device flags make the remaining kernels do no numerical work. Small transfers still synchronize and are not automatically cheap: compare their measured cost with an entirely device-controlled loop before choosing. That loop requires an ILGPU/CUDA feasibility and performance prototype, not a blanket prerequisite.
Does AMReX alignment require device-only regridding?
No. Keep tagging and numerical field remapping on the GPU, while allowing bounded compact tag/layout metadata for host clustering and allocation. AMReX itself uses host coordination in GPU tag collation. Fully device-driven clustering and regridding are later optimizations to justify with measurements.
Gate: new or migrated numerical paths have no bulk field downloads or CPU completion of reductions. Audit direct and indirect copies, metadata transfer counts/bytes, synchronization time, launch overhead, peak memory and end-to-end performance. Count requested output separately. Scientific checks, precision and existing performance limits remain binding.
GPU-01 scope, source inventory and acceptance · Algorithm execution details.
08 / Numerical review
Follow an idea all the way
to its equations and evidence.
The algorithm companion starts with the physical model and opens into discrete equations, source code and validation protocols. Current behavior, proposed algorithms and unresolved derivations are labeled separately.
- Start with physics and field locationsIncompressible flow, units and staggered control volumes.
- Trace today's complete time stepForces, diffusion, pressure signs and continuity.
- Inspect today's transport equationsBacktracing, interpolation, correction and limiting.
- Derive a conservative updateShared transfers and an illustrative scalar algorithm.
- Identify the work still to deriveStaggered momentum, cut volumes and moving walls.
- Review the proof and test obligationsConservation, stability, accuracy and physical validation.
09 / Delivery gates
Prove each step before adding the next.
Each effort follows the execution policy for the work it changes. AMR-02 does not require all of GPU-01 to finish; TRN-01, S2 and S3 are not blocked by unrelated migration gaps. New numerical paths must keep their work resident, document bounded host coordination and preserve the applicable scientific, memory and performance gates.
- AMR-02
Align today's structures
Inventory consumers, introduce canonical descriptors, migrate ownership and callers, then remove superseded representations.
Gate: existing inputs, fields, measurements, outputs and scientific acceptance remain valid. No new mesh capability, bulk field transfers or abstraction-driven performance/memory regression. Existing GPU-01 gaps remain explicitly tracked.
- TRN-01
Validate selectable single-grid transport
Measure current budgets, test shared scalar fluxes, formulate MAC momentum transport, then qualify boundaries, cut cells and separately supported motion. Integrate one recorded method choice per run with explicit support, timestep and memory requirements.
Gate: conservation and stability checks pass, retained methods preserve their scientific gates, and preselected cases demonstrate candidate accuracy benefit with acceptable cost. Selection is reproducible and unsupported configurations are rejected.
- S2
Connect patches at one resolution
Begin with the same problem split in two. Extend the accepted transport method with shared fluxes and same-level sampling, plus coupled pressure/diffusion, geometry, measurements and output.
Gate: partitioned and monolithic solutions and momentum budgets agree within declared tolerances, including periodic and cut interfaces. A simple two-patch test alone does not finish S2.
- S3 / fixed
Add resolution in one region
Start with one coarse base, one aligned 2:1 fine patch and synchronized timesteps. Add coverage masks, transfers and composite equations.
Gate: accuracy and conservation pass against uniform-fine references, with less measured memory or time at comparable measurement accuracy.
- S3 / adaptive
Let refined regions change
Extend fixed refinement to multiple patches/levels. Then add GPU tagging and safe field transfer, with bounded compact metadata for host layout management and allocation, plus geometry/solver rebuilding.
Gate: refine/derefine cycles preserve the required budgets and constraints; metadata transfers and regrid costs are measured. Thermal and moving-geometry combinations need their own evidence. Fully device-driven regridding and time subcycling are later decisions.
S2 layout work can overlap transport development. Its transport interfaces follow TRN-01's established method. Coordinate scalar work with T1, while keeping thermal acceptance separate.
Continue reading
The design behind the diagrams.
Algorithm companionDetailed numerical referenceData-model alignmentSingle-level transportS2 and S3 implementationRoadmapCurrent task status
The Markdown plans define scope and acceptance. The work board owns current status. Proposed type names and diagrams are explanatory; they are not executable API or file-format specifications.