sots-re/campaign/rollout/gate-followup.md

4.5 KiB

Astra integration followup — repair gate/report acceptance gaps

First session delivered useful initial tooling but incomplete enforcement. Same file ownership, fresh bounded session openai/gpt-5.6-terra. Read independent-review.md if present and your checkpoint. No delegation, commits, live lab operations or architecture changes. This is a precise Astra approved repair contract, not permission to waive any of the initial requirements.

Fix these inspected defects and write adversarial tests proving them:

  1. corpusNonzero cannot be absence of "0 save" in ctest --output-on-failure. Parse per-test JUnit system-out positive summary counts for all four corpus tests; require count equals manifest top-level saves count (tests scan top-level, do not hash nested unexecuted .sav as exercised). Missing summaries, zero/mismatched count and duplicates fail. JUnit executed identity multiset must exactly match discovered/expected inventory (including tests absent from XML -> fail).
  2. Source copy integrity: compare every copied byte hash/mode to pre-copy snapshot BEFORE build. startswith('build') excludes actual source filenames like build_turn.cpp. Exclude only top-level known build output directories or explicitly named runtime artifact directories. Track deleted files and symlink semantics or reject source symlinks explicitly. Snapshot includes untracked files.
  3. Require tools versions and complete hashes for actual supplied SOTS_* file/dir dependencies, assets and trace inputs. Explicitly sanitize inherited SOTS_* environment or validate/hash each consumed input. Record roots, actual env settings, compiler/build settings and skip limitations.
  4. --shim should be a boolean flag; use source cmake/toolchain-mingw-i686.cmake. Full requires --data and shim build; --shim on host also builds. Check resulting DLL hash. Initial --shim PATH requires directory but passes directory to CMAKE_TOOLCHAIN_FILE, unusable. Correct docs/help/tests.
  5. RE identity mutation: don't hash transient campaign runtime/publication state as executable source. Bind actual RE tooling inputs via immutable copies or hash list before+after; list excludes only runtime/evidence output areas explicitly. Engine source must be stable before+after. Gate should run while other independent campaign checkpoints change without falsely claiming tools are mutable. Detect actual gate/checksum/inventory source mutation.
  6. run/errors: subprocess exceptions need failed manifest, expected actual statuses must hold, input roots/output roots must not overlap, jobs positive, binary must exist for passing gate. All required checks explicitly present, not all(empty) success. all over strings is unsafe.
  7. Standalone reporter: verify passed gate schema and binary hash/source identity thoroughly. Preserve full stderr/stdout and complete paths/diffs. --roundtrip only if CLI semantics means verify then SIMULATE (inspect src/app/main.cpp; don't accidentally test identity instead of turn). Distinguish accepted ON SELECTED WORKLOADS from campaign replacement acceptance; reporter is measurement by default and --accept file equality criterion explicitly documented. Hash all engine-arg file inputs (commands/data) actually used, prevent extra args overriding --out/metric or injecting --roundtrip-only/etc. Detect binary/input mutation after run too. Keep required coverage and equality separation. Add negative-path tests missing pairs, binary mismatch, malformed/failed manifest, missing coverage, pair failure, wrong reference and unexpected args.

Do not rerun integrated gate until lead says source stable. Unit tests and fixture negative paths now; checkpoint exact manifest/CLI shape and results. Main will run fresh final gate later.

Astra resolution of independent review R2

The equality reporter must NOT claim campaign acceptance. Replace --accept with --require-match (no legacy alias). This flag only requires all selected pairs to match and affects exit status; report status remains measured on a valid measurement, including when equality holds. Add matchRequired and allPairsMatch booleans and an explicit scope statement. A failed required match has status failed and retained residual evidence. ONLY campaign controls + independent verdict/integrated criteria can produce accepted status. This removes no-op-as-accepted semantics without pretending a general equality tool can infer whether a workload exercised research. Test identical input/oracle can be measured but never gets accepted; full gate source validation still required. Pilot provides required positive phase/branch controls separately.