# Performance regression checks

**GPU policy revised 2026-09-26; runtime migration pending.**
[GPU-01](/docs/plans/gpu-resident-processing) prioritizes resident fields, complete device
reductions and removal of bulk-field CPU round trips. Host setup and explicit bounded
control/status/layout metadata are allowed; record their payload, count, cadence,
synchronization and memory cost separately from requested output. Audit direct and
indirect transfers across setup, steps, rebuilds and failures, exposing instrumentation
blind spots. Zero copies alone is not a performance result: excessive inactive launches
or speculative buffers can be worse than measured status polling. AMR-02 does not
require full GPU-01 closure, but must introduce no new bulk numerical readbacks and
meet the unchanged [architecture acceptance](#architecture-change-acceptance) gates.
This revision changes none of the timing/reference rules or limits below.

All three `verify-all` selections (`quick`, `multigrid`, `full`) measure the same nine mandatory cases: two planar
Taylor–Green schemes, four exact periodic controls, two short moving fans and the
accepted backward step. Numerical acceptance and timing acceptance remain separate.
See [complete validation](/docs/COMPLETE_VALIDATION) for the full catalog and resume contract.

```sh
dotnet build PolyCfd.sln -c Release
# Intentionally accept a new reference only after all quick checks pass.
dotnet run --no-build -c Release --project src/PolyCfd.Cli -- verify-all --suite quick --backend cuda --output output/validation/record-timing --record-performance --performance-baseline output/references/performance-raw.json
# Compare on the same idle device, without rewriting the reference.
dotnet run --no-build -c Release --project src/PolyCfd.Cli -- verify-all --suite quick --backend cuda --output output/validation/check-timing --check-performance --performance-baseline output/references/performance-raw.json --max-slowdown-percent 20
```

Supply the reference explicitly. Missing clock/coverage/reference compatibility is
incomplete or failed coverage, never a measured performance pass. Recording requires
a new reference path and every selected check to pass. Timings and comparisons live
inside the `performance/attempt-NNN/` directory under the suite output.
The old stopwatch `Baselines/performance.json` remains historical evidence; do not
rescale it or compare it directly with the new raw-clock boundary.

## What is measured

The suite installs a validation observer; production time integration is unchanged.
On Linux it uses **CLOCK_MONOTONIC_RAW**, required by this WSL workstation's observed
clock-rate changes. Other platforms use `Stopwatch.GetTimestamp` and record that clock.
The clock and timing boundary must match the reference exactly.

- **Whole verification call:** setup, geometry/cache construction, integration,
  diagnostics, output and scientific acceptance. Excludes process startup, builds,
  suite bookkeeping and the later fan time-series audit.
- **Median solver step:** synchronized integration, including adaptive selection and
  moving geometry. Host output/diagnostics after the step are excluded. The first
  two steps are warm-up; at least three remaining samples are required.
- **Device and environment:** actual requested accelerator, machine, runtime,
  configuration, processor/thread and cuBLAS settings. No silent backend fallback.

Internal `ValidationResultBase.Performance` still describes its existing stopwatch
boundary and remains in saved verification records. The suite's separate performance
artifact is authoritative for raw-clock comparisons; do not mix those measurements.
The preserved incremental-projection benchmark uses this same raw boundary and nine
case keys, permitting a direct before/after check without replacing its reference.

A metric fails only when **both** conditions hold:

```text
current > reference × (1 + maxSlowdownPercent / 100)
current − reference > absolute noise floor
```

The noise floor is 1,000 ms for elapsed time and 1 ms for median solver-step time.
For example, with a 20% allowance, an elapsed increase from 10 to 13 seconds fails;
1 to 1.3 seconds does not. Equality at the allowance or noise floor passes.

## Reference lifecycle and comparable workloads

`PerformanceSnapshot` records a format version, UTC timestamp, environment,
correctness result, and case measurements. Environment matching covers machine
name, OS, process architecture, logical processor count, .NET runtime, build
configuration, and `POLYCFD_MAX_THREADS` / `POLYCFD_ENABLE_CUBLAS` settings. Device
names must match per case. An environment or accelerator mismatch fails with an
explanation; it is not reported as a measured slowdown.

The nine mandatory cases have stable keys. Saved-case checks are separate from
this timing catalog. Adding or removing a case
requires deliberately recording an updated reference; missing coverage cannot
silently make the suite pass. Saved-baseline dispatch requires canonical scenario and verification records, checks resolved inputs
and reference policy, and permits current-build comparison. Complete older references need explicit
lossless conversion; missing inputs are rejected. Archived CSV-only evidence is reported separately.
See the [runtime contracts](/docs/RUNTIME_CONTRACTS) and [validation guide](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/validation/PolyCfd.Validation/README.md).

Only explicit `--record-performance` creates a new reference, and only if all
selected checks pass and measurements are valid. Existing references are preserved. A correctness failure or a comparison failure
does not update the reference.

Every completed mandatory case stores its measurement in `report.json`, including
when a later case fails. Once all nine mandatory cases pass, the `performance`
check writes `performance.json` and, when a compatible reference is supplied,
`comparison.json` inside its attempt directory. `ValidationPassed` describes
numerical correctness; the comparison and suite report give the separate timing
verdict. The reference path is never used for a latest-report side effect.

Timings remain sensitive to concurrent CPU/GPU work, clock/power settings, thermal
throttling, and driver changes. Recheck an unexpected slowdown on an idle machine.
Do not refresh a reference merely to suppress a regression. Dated numbers in
[timing.md](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/docs/timing.md) are historical logs, not portable thresholds.

The [backward-step OpenFOAM comparison](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/docs/BACKWARD_STEP_PERFORMANCE.md) is a separate
cross-code benchmark with external process wall times. It does not replace these
local regression references or use OpenFOAM timings as a `verify-all` threshold.

## Initial reference — 2026-09-16, before backward-step

The initial local Release run of `verify-all --record-performance` passed all 11 checks
on the NVIDIA GeForce RTX 3080: both mandatory fan advection schemes and nine
saved cases. It exited 0 and saved `Baselines/performance.json` and
`Baselines/performance.latest.json`. Each case has valid elapsed and median-step
measurements; the reference records the machine, runtime, configuration, and device.

The complete log, reference copies, per-case timing table, build/test results,
and cylinder-reference migration evidence are saved locally under
`output/validation/performance-record-2026-09-16/`. The original cylinder references
were preserved before adding measured force histories, and both replacements
matched their original wake tolerances. The fine cylinder also passed its
canonical force checks during the full suite. No numerical tolerances were relaxed.

This run established the first timing reference. Recording does not itself test
for a slowdown against an older reference. Adding the backward-step case requires
a new recording because performance case coverage must match.

## Source and regression coverage

- [SimulationPerformance.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/validation/PolyCfd.Validation/Performance/SimulationPerformance.cs): timing result and warm-up/median recorder.
- [PerformanceBaseline.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/validation/PolyCfd.Validation/Performance/PerformanceBaseline.cs): snapshot format, validation, atomic recording, and comparison.
- [VerificationRecord.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/validation/PolyCfd.Validation/Scenarios/VerificationRecord.cs) and shared scenario records: explicit numerical workload and instrumentation settings.
- [ScenarioCaseRunner.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/validation/PolyCfd.Validation/Framework/ScenarioCaseRunner.cs): synchronized timing capture around the shared executor.
- [SuiteCommand.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/src/PolyCfd.Cli/SuiteCommand.cs): CLI coverage, performance comparison and exit status.
- [SuiteRunner.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/validation/PolyCfd.Validation/Suites/SuiteRunner.cs): durable attempts, integrity checks and resume.
- [SuiteCases.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/validation/PolyCfd.Validation/Suites/SuiteCases.cs): scientific cases and raw-clock observer.
- [PerformanceBaselineTests.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/test/PolyCfd.Tests/Unit/PerformanceBaselineTests.cs): thresholds, noise floors, invalid data, environment/device mismatches, coverage changes, and reference preservation.
- [MovingFanValidationTests.cs](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/test/PolyCfd.Tests/Unit/MovingFanValidationTests.cs): real fan stability and populated timing measurements.

The automated unit tests use deterministic synthetic timing values for pass/fail
rules. `dotnet test` does not impose machine-dependent speed limits. CLI checks
also exercised real fan/Taylor–Green recording and comparison, a deliberately
faster reference that forced exit 1, and refusal to refresh after a failed case.

The mandatory `backward-step-regression` now contributes a 384x32x32, Re_h=100
MacCormack run at dt=0.05 through t=120. It checks physical accuracy against an
embedded OpenFOAM reference before performance can be recorded. Existing timing
references need an explicit `verify-all --record-performance` refresh to include
this new case; case-coverage changes are never silently ignored. See
[Backward-step validation](/docs/BACKWARD_STEP_VALIDATION).

## Expanded reference — 2026-09-16

The completed Release `verify-all --record-performance` run passed **12/12**:
two moving-fan checks, the new backward-step check, and nine saved baselines.
It saved `Baselines/performance.json` and `Baselines/performance.latest.json`.
The step case took **202.61 s**, with **82.76 ms** median solver-step time
(2,398 timed steps after two warm-up steps). Summed simulation elapsed time for
the suite was 2,021.50 s, about 33 minutes 42 seconds.

An audit against the previous reference passed the existing 20% gate for all
11 common cases. The largest elapsed-time increase was 8.14%, and the largest
median-step increase was 5.86%; both were the cavity-with-sphere case. The new
step is additional coverage, not an existing-case slowdown.

The log, previous/new reference copies, and timing audit are preserved under
`output/validation/backward-step/`. The accepted performance file SHA-256 is
`4f16c2f9c9bae049b004e6a3e491d8fcfbc6a3b8b8102b47961773aa1bc084fc`.

## Simplification acceptance and instrumentation

The [implementation evidence](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/docs/ARCHITECTURE_IMPLEMENTATION_EVIDENCE.md) records cumulative CUDA acceptance, rejected intermediate measurements and deferred optimizations. Run GPU comparisons sequentially. Stage profiling is enabled by default; match that policy across comparisons (`--no-stage-profiling` disables it). Memory-audit counters describe logical application transfers/allocations, not physical PCIe traffic or hidden ILGPU work. Keep audit settings identical and disabled for strict timing comparisons.

## Architecture change acceptance

The architecture refactor used a stricter criterion than the CLI allowance: no repeatable regression outside measured run-to-run noise for each affected representative workload. Aggregate improvements do not excuse a slower affected case. Compare repeated pairs on an otherwise idle machine against the original and immediately preceding accepted revisions; three pairs are a starting point, not automatic proof. Gather more evidence for noisy or close results. A zero-percent CLI allowance still retains the absolute noise floors above, so inspect raw elapsed and median-step measurements as well as the exit code.

Match grid, geometry, solver, tolerances, timestep policy, final time, required output fields/cadence, diagnostics, profiling mode and accelerator. Record revision, runtime, driver, build, device and relevant environment settings. Separate setup, stepping, geometry rebuild and output costs; audit H2D/D2H/D2D bytes and calls, allocations, peak retention and synchronization independently of final uninstrumented timings. Viewer changes also need comparable latency, decode, upload and retained-memory measurements. Logical payloads and CPU render submission times are not physical bus traffic or GPU completion timings.

Preserve the [transfer and memory budgets](https://github.com/hankbeasley/polycfd/blob/2515fae75250ee310a783c96057bc4b0a4fa9ee6/docs/ARCHITECTURE_IMPLEMENTATION_EVIDENCE.md#transfer-and-memory-budgets): no extra steady-state uploads of unchanged geometry/BCs, duplicate same-state captures, or full-volume readbacks for scalar-only consumers. Keep required numerical checks and useful output. Revise or defer a change that fails its budget or timing comparison; ambiguous evidence leaves acceptance pending. Do not silently refresh references or infer local regression acceptance from an external OpenFOAM comparison. When suite coverage changes, compare common workloads separately before explicitly accepting a new reference.

## WSL raw-clock method comparison

On this workstation, ordinary monotonic/calendar clocks have shown rate changes.
The [incremental-pressure timing harness] (unavailable local reference: `../../openfoam/compare/projection_benchmark/README.md`)
uses `CLOCK_MONOTONIC_RAW` for both synchronized steps and whole verification calls,
with explicit CUDA and identical instrumentation on preserved Release builds. It
exercises the same nine mandatory workloads and their existing physical gates; a
separate cavity key adds the 128³ short trajectory. Its unchanged 20% comparison
uses the existing 1 ms step / 1,000 ms elapsed noise floors.

These measurements are separate from the verifier's existing Stopwatch timings,
which remain in their original artifacts. Whole-call timing includes setup, output
and acceptance but excludes process startup/build; standalone enclosure process
timing also includes startup. Do not mix these boundaries or overwrite an accepted
baseline. See [the method validation report] (unavailable local reference: `../../openfoam/compare/INCREMENTAL_PROJECTION.md`).

## Chebyshev diffusion quick comparison — 2026-09-26

The final SOL-02a build (frozen as `output/builds/sol02a` until SOL-02c removes it; assembly hashes
in the report's provenance) was compared with the pre-change quick reference
`output/validation/sol04a-quick/performance-reference.json` in
`output/validation/sol02a-fix-quick/performance/attempt-001/comparison.json`, on an idle machine (GPU
0% before and 1% after, no other PolyCFD or test processes; `output/validation/sol02a-fix-idle-checks/`).
The check passes with no issues. Median solver step: fan-sl 296.77 → 111.16 ms (−62.5%), fan-mc
310.23 → 119.36 ms (−61.5%), backward step 59.63 → 57.86 ms (−3.0%), periodic and planar cases +0.2%
to −5.9%. Elapsed: fans −9.1% and −9.5%, backward step −2.7%; beltrami +44.9%, planar-sl +19.2% and
planar-mc +12.8% are small absolute changes (under a second) inside their noise floors. The run on the
build before the review fix pass (`output/validation/sol02a-quick/`) passed the same way. The multigrid
and full-suite comparisons against the WSL reference follow in SOL-02c.

## Diffusion fallback measurements — 2026-09-24

The pre-fix `eee86ac` build and the diffusion fallback candidate were frozen
separately before measuring the same nine scientific workloads on CUDA. Both
complete pairs pass scientific acceptance. Verification-call totals are
276.466 → 294.795 s (+6.63%) and, in reversed order, 263.718 → 264.555 s
(+0.32%). Combined change is +3.55%, with substantial run-to-run variability.
The first pair's shear-Y median misses the unchanged 20% allowance; the reversed
pair passes. Neither that failure nor the earlier unchanged-build historical
shear-Y failure (+22.2%) was replaced or used to refresh a reference.

Two-order isolated diffusion tests (50 warmups, 200 samples, independent final
residual checks) show mean changes of -0.76% at 64³ and -0.84% at 128³. These
are fast-path inputs; extra work needed by formerly failed enclosure solves is
not covered. The initial full suite stopped on the historical gate (backward-step call +25.1%, median +23.7%; shear-Z +27.7%; planar-SL +21.2%). It passed all 1,519 selected tests and nine scientific cases first. A subsequent small ordinary-path cleanup passes 92 focused tests, four startups and a complete matched timing pair (process wall time 316.324 → 288.697 s). Balanced isolated diffusion samples change by about -1.6% at 64³ and +1.0% at 128³. These variable observations do not establish a speedup. Full validation completed in `output/validation/diffusion-full-verified-20260924`: all scientific checks pass, while the historical gate remains failed after one fresh timing retry (backward-step call +32.9%, median +25.8%; shear-Z and planar-SL medians +25.8%). A control using all 26 unchanged historical assemblies and matching scientific identities also fails the old gate (backward-step call +41.8%, median +24.9%). Its verification calls total 312.693 s versus 293.330 s for the current retry. This contemporaneous comparison passes the unchanged comparison limits, but includes intervening code changes. A five-second monitor captured short-lived Windows PolyCFD workers during both sequences; these are not isolated timings. Latest-sample checks had missed those bursts. Coordinate a quiet test session before another deliberate phase; do not rebaseline or repeat until a favorable sample appears. Commands, identities, raw
clock boundaries and every timing attempt are linked in the
[maintained report] (unavailable local reference: `../../openfoam/compare/ENCLOSURE_N128_DIAGNOSTICS.md`) and
[JSON] (unavailable local reference: `../../openfoam/compare/enclosure_diffusion_fix.json`).

A deliberate `verify-all --resume --retry-failed` performance retry now reruns
all nine scientific workloads in a fresh attempt, preserving the initial timings,
reference and completed long checks. The unchanged allowance and scientific
acceptance apply to every fresh measurement. See
[restart semantics](/docs/COMPLETE_VALIDATION#results-and-restart).