Gauntlet Test Suite Audit — 2026-09-02

Per-test classification inventory for the gauntlet suite and its cross-OS harness. This document is the decision input for cleanup and pipeline changes; it recommends but does not apply them.

Scope

"Gauntlet" names three separate things in this repo. All three are inventoried:

Surface	What it is	Runs in CI?
tests/gauntlet/**	Vitest suite. pnpm gauntlet (9 tests) and pnpm gauntlet:evals (4 tests).	score.test.js yes, via run-vitest-bind-aware.mjs root. evals.test.js no — excluded by design (LLM/wall-clock spend).
.local_testing/**	Docker harness named "gauntlet" — run-gauntlet.sh / .ps1, three Dockerfiles, test-base-commands.sh.	No. No workflow references it.
scripts/cli-smoke-parity.mjs SMOKE_CASES	The 8 cases the smoke-macos / smoke-windows / smoke-parity jobs execute.	Yes — this is the cross-OS pipeline stage.

Classification legend

• (a) Valid, cross-OS relevant — exercises a live code path; either platform-independent or a genuine OS-divergence detector.
• (b) Valid but platform-specific, questionable signal — runs, but its signal is weak or duplicated elsewhere.
• (c) Stale — targets a code path or feature that has changed or been removed.
• (d) Too niche — a real edge case, but unlikely to regress in a release.

Method

Every case was read against the code it exercises. Both vitest suites were executed on Windows 11 / Node 22 on 2026-09-02: pnpm gauntlet → 9 passed; pnpm gauntlet:evals (after pnpm build) → 4 passed. The .local_testing Dockerfiles were not built — Docker was unavailable in the audit session, so base-image tag validity is recorded as unverified rather than asserted.

---

A. tests/gauntlet/sourcevision-evals/score.test.js — 9 cases

Pure-function tests over score.js. No filesystem, no spawn, no path handling — platform behaviour cannot influence them, so all are cross-OS relevant in the trivial sense.

#	Case	Class	Rationale
1	archetypeAccuracy returns 1.0 on exact match	(a)	Baseline identity of a scorer the eval gate's pass/fail depends on.
2	returns 0.5 when half the archetypes are perturbed	(a)	Pins the scorer's linearity; a scorer that saturated to 0 or 1 would still pass case 1.
3	ignores files present only in actual (denominator = intersection)	(a)	Encodes the documented decision that adding fixture files must not drag the score down.
4	treats null and missing archetype as equivalent	(a)	Live contract (sv emits unclassified files), but the title overclaims — see Defect D1.
5	zonePartitionSimilarity returns 1.0 on exact partition match	(a)	Baseline identity of the second scorer.
6	survives zone-id relabeling	(a)	The scorer's whole reason for existing: zone IDs are LLM-generated and unstable, partitions are not. Directly protects the gate from false failures.
7	returns < 1 when a golden zone splits across actual zones	(a)	Pins Jaccard behaviour on the most likely real regression (a zone fracturing). Assertion trio is redundant — see Defect D2.
8	returns 0 when zones share no files	(a)	Floor of the range; cheap.
9	handles empty-on-both-sides as perfect match	(d)	Guards the golden.zones.length === 0 early return. Failure probability in a release is near zero: no fixture has zero zones, and the branch is two lines. Retained on cost grounds only — it is one line of test for one line of guard.

Applied 2026-09-03 (cleanup task). Case 9 was not deleted but generalised: it is now a describe("degenerate inputs") block covering both scorers across six empty and non-overlapping input combinations instead of the single one it asserted. Recommendation 2 below said to keep it as-is on cost grounds; broadening it proved strictly better, because that input class is exactly where archetypeAccuracy hides a live hazard — see the correction to section B. Case 4 was rewritten to actually reach the ?? null branch (D1), case 7's two subsumed assertions were dropped (D2), and projectForScoring gained four cases (D3). score.test.js: 9 cases → 24.

Not covered: projectForScoring — exported from score.js, consumed by both evals.test.js and record-goldens.js, and has no unit test. See Defect D3.

B. tests/gauntlet/sourcevision-evals/evals.test.js — 4 cases

Spawns the built sv analyze --fast per fixture and scores against a committed golden.

#	Case	Class	Rationale
1	toy-app archetype accuracy ≥ 1.0	(a)	Green today at floor 1.0; 8 files across 8 distinct archetypes is real coverage of the classifier.
2	toy-app zone partition similarity ≥ 1.0	(a)	Pins Louvain output on a small graph where a parameter change is visible.
3	medium-app archetype accuracy ≥ 1.0	(a)	31 files / 13 archetypes — the only fixture large enough to catch a heuristic that regresses only at scale.
4	medium-app zone partition similarity ≥ 1.0	(a)	7 zones with a server/client/shared split; the realistic zone-detection regression target.

Cross-OS note — this suite is a real OS-divergence detector. Goldens store paths with POSIX separators (src/client/App.tsx) and the scorers compare path strings byte-exactly as Map keys. The gate passed on Windows in this audit, which proves sv analyze normalises separators in classifications.json and zones.json. A regression that emitted src\client\ App.tsx on Windows would drop both scores to 0. That property is not tested anywhere else.

Correction, 2026-09-03. The last sentence but one is wrong: the two scorers do not fail alike, and only one of them catches that regression. On a total separator mismatch every actual path misses the golden Map, so archetypeAccuracy never increments total, hits its total === 0 ? 1 guard, and returns a perfect 1.0 — clearing its floor. Measured directly: archetypeAccuracy 1.0, zonePartitionSimilarity 0. So cases 1 and 3 above (the archetype floors) contribute nothing to this property, and the gate detects a Windows separator regression through cases 2 and 4 alone.

This is why case A#9's (d) classification was the wrong call to act on literally: the empty / non-overlapping input class it guarded is not a near-zero-probability edge case, it is the shape a real separator regression takes. It is now pinned explicitly in score.test.js ("degenerate inputs") and documented in evals.test.js and the harness README, so a future maintainer cannot quietly weaken the zone floors on the assumption that the archetype floors back them up. The gate is not hardened here — the behaviour is only pinned and documented; see follow-up 6 below.

On the sv-version gap: goldens record svVersion: "0.1.0"; sourcevision is now 0.5.1. That gap is not staleness — the gate re-runs the analyser live and still scores 1.0, so the algorithmic output has been stable across four minor versions. It does mean the gate has never been the thing that caught a regression, because nothing runs it automatically.

C. .local_testing/test-base-commands.sh — 4 cases

Bash-only smoke script. Every case asserts exit code 0 and nothing else.

#	Case	Class	Rationale
1	ndx config succeeds	(b)	Exit-code-only; tests/e2e/cli-config.test.js covers the same command with output assertions, on both macOS and Windows in CI.
2	ndx init --provider=claude --model=claude-3-5-sonnet-20241022	(c)	Stale argument. claude-3-5-sonnet-20241022 is absent from packages/core/llm-model-catalog.js, which now lists claude-sonnet-5 and claude-opus-5. validateInitFlags (packages/core/init-llm.js:604-615) emits an "Unknown model" warning and proceeds, and the case discards both streams to /dev/null — so the test passes while silently exercising a retired model ID. The flags themselves (--provider=, --model=) are still live.
3	ndx analyze . succeeds	(b)	Exit-code-only; duplicated by tests/e2e/cli-orchestration.test.js and the sourcevision package suite.
4	ndx status . succeeds	(b)	Exit-code-only and strictly weaker than smoke-parity cases status-missing-rex / status-json, which assert error codes and a JSON contract on both OSes.

Applied 2026-09-03 (cleanup task). Case 2 was repaired, not removed. Its concern — that ndx init accepts --provider= / --model= — is live, and the task's own guardrail is not to delete a test whose concern is valid but whose implementation is narrow. The stale model ID is gone (claude-sonnet-5, hoisted to an INIT_MODEL variable with a pointer to the catalog), and the case no longer discards both streams: it greps the captured output for Unknown model and fails on a hit. That closes the loop that let it rot silently — verified against validateInitFlags, which warns on the old ID and is clean on the new one. Cases 1, 3 and 4 are untouched: they are (b) weak-signal, not stale, and the same guardrail applies.

The three (b) cases and the harness defects H1–H7 are not addressed here. Recommendation 1 offers "retire the directory, or fix H1–H7 as a set"; both are decisions about a developer-facing workflow that README.md, CONTRIBUTING.md and packages/core/README.md document as supported, not test-case cleanup, and two completed PRD features built it deliberately. Tracked as follow-up 7 below.

Structural finding: none of these four run under the documented primary entrypoint. run-gauntlet.sh hardcodes test_command="pnpm test" (line 160) and overrides the image CMD, so test-base-commands.sh executes only via docker compose run ndx-macos or a manual docker run. The README's platform table claims macOS runs "pnpm test + /ndx/test-base-commands.sh"; no configured path does both.

D. scripts/cli-smoke-parity.mjs SMOKE_CASES — 8 cases (the cross-OS pipeline stage)

Collected on macOS and Windows, compared by smoke-parity. Semantics are documented in docs/contributing/cli-smoke-parity.md; the classifications below assess parity value specifically — what the cross-OS comparison adds beyond running the case on one OS.

#	Case	Class	Rationale
1	version-text — ndx version	(b)	Reads packages/core/package.json and prints a string. No path, temp dir, or child process in the compared output; parity cannot diverge unless Node itself does. Valuable as a smoke check, near-zero as a parity check.
2	version-json — ndx version --json	(b)	Same as #1 plus a one-key JSON contract.
3	unknown-command — ndx foobar	(b)	Compares only failure.code (projectComparableForParity discards detail). The code is a constant in the CLI's error path — OS-independent by construction.
4	typo-suggestion — ndx statis	(d)	Edit-distance suggestion over a static command list. Platform-independent, and the failure mode (suggestion list drifts) is caught by tests/e2e/cli-hints.test.js on every OS. Lowest-value case in the set. Removed 2026-09-03 — see below.
5	help-rex — ndx help rex	(b)	Static help text; substring assertions. Parity adds nothing the ubuntu validate job doesn't already cover.
6	plan-help — ndx help plan	(b)	Same as #5.
7	status-missing-rex — ndx status <empty tmpdir>	(a)	One of only two cases touching a real filesystem path. Exercises mkdtemp + missing-.rex detection, where Windows path and permission behaviour genuinely differs. Parity compares NDX_CLI_NOT_INITIALIZED across OSes.
8	status-json — ndx status --format=json <seeded .rex>	(a)	Highest-value case in the set: seeds a real .rex fixture via setupRexDir, reads the PRD folder tree off disk, and compares a full nested JSON tree — including item ordering — across OSes. Directory-read ordering and CRLF handling are real Windows divergence risks, and this is where they surface.

Applied 2026-09-03 (cleanup task). Case #4 (typo-suggestion) removed — the only test deleted outright in the cleanup. Its three assertions (NDX_CLI_UNKNOWN_COMMAND, Did you mean, status) are made verbatim by tests/e2e/cli-hints.test.js:19-25, which run-vitest-bind-aware.mjs root executes in the validate (ubuntu), smoke-macos andsmoke-windows jobs — one platform more than this collector reaches, so deleting it widens OS coverage of the concern rather than narrowing it. A cross-reference comment marking that file as the receiving suite is committed alongside. unknown-command (#3) retains an error-path shape fingerprint, so the shape check still covers a failure stream; the path-free shape set goes 6 → 5 and the sequence 8 → 7 cases.

Parity-coverage limitation, worth knowing before adding cases. normalizeText (scripts/cli-smoke-parity.mjs) rewrites \r\n → \n and every \ → / before anything is compared, and stripKnownRuntimeNoise removed the [child-lifecycle] process group cleanup is not supported on this platform line. The doc frames this as ignoring "expected OS-specific differences such as temp paths, shell wording"; the consequence it does not state is that a genuinely wrong separator or line ending in user-facing output is normalised away too. Parity therefore detects semantic drift, not path or line-ending drift. Line-ending behaviour is covered separately by tests/e2e/prd-line-endings.test.js.

Addressed 2026-09-02. Cases #1–#6 now also carry a shape fingerprint — \r\n and \ counts taken on the raw stream before normalizeText — which is compared across OSes. #7 and #8 stay excluded: they print the temp directory, so their native backslashes are correct rendering, not drift. See docs/contributing/cli-smoke-parity.md.

Second finding, addressed 2026-09-03. The [child-lifecycle] strip named above was not runtime noise at all — it was n-dx's own output, stripped as a workaround while a real Windows regression was live (the notice printed on every win32 invocation, including ndx version; fixed in b0efffdd / #329 by gating it behind NDX_DEBUG_LIFECYCLE). Retaining the strip after the fix left version-text's stderrExact: "" baseline structurally unable to catch the one regression class in this stage's subject matter for which there is concrete historical evidence. The strip is now removed and the boundary is explicit: strip what the Node runtime writes, never what n-dx writes. Two replay cases in tests/unit/cli-smoke-parity.test.js pin it, and the live Windows collector was re-verified green without the strip (Windows 11 / Node 22, 2026-09-03).

Net read for the sibling task ("does the final cross-OS step earn its CI cost"): 6 of 8 parity cases (#1–#6) compare output that contains no OS-shaped data, so the comparison step adds no signal for them — the per-OS compareExpected baseline check inside each smoke job already covers what they assert. Cases #7 and #8 are the two that justify the stage. The comparison job itself (smoke-parity) is cheap: ubuntu, 10-minute timeout, two artifact downloads and one script run. The cost sits in smoke-macos (macOS bills 10× Linux) and smoke-windows, which run the test suites regardless of whether the parity comparison exists.

Correction, 2026-09-02. "the per-OS compareExpected baseline check inside each smoke job" was wrong when written: compareExpected ran inside smoke-parity, two jobs downstream, so a per-OS contract break reddened ubuntu rather than the platform that broke it — and went unenforced entirely whenever either smoke job failed, since smoke-parityneeds both. The sibling task's review (docs/contributing/cross-os-pipeline-review-2026-09.md, defects S1/S2) found this and moved the baseline into collect. The sentence is accurate now.

---

Harness defects found

Not test cases, but they determine whether the cases can run at all. Listed with evidence.

ID	Where	Finding
H1	.local_testing/run-gauntlet.sh:63-64	On a Linux host, detect_platform returns linux, so DOCKERFILE resolves to Dockerfile.linux — which does not exist. There is no linux → macos mapping and no existence check before docker build -f "$DOCKERFILE". ./run-gauntlet.sh on Linux fails at build. The README (line 35) documents Linux as supported via Dockerfile.macos.
H2	.local_testing/run-gauntlet.sh:14, 235-264	--platform=<os> is documented in the script's own show_help and in the README (line 37), but the argument parser has no --platform=* branch — it falls through to *) and exits 3 as an unknown option. Even if parsed, DOCKERFILE is computed at line 64, before parsing, so a later PLATFORM change would not take effect.
H3	.local_testing/.dockerignore	Inert. Both Dockerfiles use build context .. (project root), and Docker reads .dockerignore from the context root — there is no root .dockerignore in this repo. Worse, the file is self-contradictory if it were honoured: it excludes scripts (but pnpm test is node scripts/run-all-tests.mjs), tests/fixtures (the eval goldens), and .local_testing (which Dockerfile.macos:33 copies from). Moving it to the context root as-is would break every build.
H4	Dockerfile.macos:4, Dockerfile.windows:4, Dockerfile.windows-nano:4	All three pin node:20-* while the repo declares engines.node: ">=22.0.0" and CI runs Node 22. The containers test an unsupported runtime, and pnpm's engine check on the workspace root is expected to fail pnpm install --frozen-lockfile outright. Base-tag validity (node:20-ubuntu, node:20-windowsservercore) is unverified — Docker was unavailable in this session, and no CI job builds these images, so nothing in the repo has exercised them since they were added.
H5	.local_testing/Dockerfile.windows-nano	Dead and misnamed. Referenced by nothing (no script, compose service, workflow, or doc), and despite "nano" it uses the same node:20-windowsservercore base as Dockerfile.windows — it is a near-duplicate, not a smaller image.
H6	.local_testing/docker-compose.yml:13, 26	Both services mount ./test-results:/ndx/test-results, but nothing writes to test-results — no script, no vitest reporter config. Dead mount.
H7	.local_testing/README.md	Three doc/implementation mismatches beyond H1/H2: claims the macOS container is "Ubuntu 20.04 LTS" (Dockerfile pins the unversioned node:20-ubuntu); claims the Windows base is "LTSC 2022" (the tag carries no OS version); and claims macOS runs "pnpm test + /ndx/test-base-commands.sh" (see section C — no configured path does both).
D1	score.test.js:44-48	Case 4's title is "treats null and missing archetype as equivalent", but both sides pass an explicit null. The ?? null coalescing in archetypeAccuracy (score.js:26, 33) — the actual "missing" branch — is never exercised. Removing both ?? null operators keeps this test green.
D2	score.test.js:82-85	Redundant assertions: toBe(0.5) subsumes the preceding toBeGreaterThan(0) and toBeLessThan(1). Harmless, but the first two contribute no signal.
D3	score.js:80-91	projectForScoring has no unit test. It is the sole adapter between raw sv analyze output and the scorers, so a shape change in classifications.json or zones.json breaks it — and score.test.js, which feeds the scorers pre-projected literals, would stay green. evals.test.js would catch it, but that suite never runs in CI.
D4	tests/gauntlet/sourcevision-evals/README.md:20, 24	Fixture descriptions are wrong. Claims toy-app has "1 zone" (golden has 3: components, hooks, src) and medium-app "6 zones" (golden has 7). Archetype counts are also understated — medium-app has 13 distinct archetypes, not the implied split.

D1–D4 fixed 2026-09-03. D1 was confirmed by mutation before the fix: stripping both ?? null operators from score.js left the old case 4 green and now fails two of the replacement cases. One correction to D4's own numbers — medium-app has 12 distinct archetypes plus one file with no archetype (13 only if null is counted as a value); the README now states it that way. H1–H7 are untouched; see follow-up 7.

Summary

Surface	(a) valid	(b) weak signal	(c) stale	(d) niche	Total
score.test.js	8	0	0	1	9
evals.test.js	4	0	0	0	4
test-base-commands.sh	0	3	1	0	4
SMOKE_CASES (cross-OS)	2	5	0	1	8
Total	14	8	1	2	25

Nothing in tests/gauntlet/** is stale — both suites are green against current code, and evals.test.js earns its keep as the only byte-exact path-normalisation check on sourcevision output. The maintenance burden sits entirely in .local_testing/, which is unreferenced by CI, has two hard-broken entrypoint paths (H1, H2), pins an unsupported Node major (H4), and whose four test cases are all weaker duplicates of e2e coverage that already runs on both OSes.

Post-cleanup state (2026-09-03)

What the cleanup task actually changed, against the classifications above:

Surface	Before	After	Action
score.test.js	9 cases	24 cases	1 (d) generalised, D1/D2 fixed, D3 added
evals.test.js	4 cases	4 cases	unchanged; path-normalisation contract documented
test-base-commands.sh	4 cases	4 cases	1 (c) repaired and made self-checking
SMOKE_CASES (cross-OS)	8 cases	7 cases	1 (d) deleted, coverage moved to cli-hints.test.js
Total	25	39	1 deleted, 1 repaired, 1 generalised, 15 added

One test was deleted (typo-suggestion), one repaired (test-base-commands.sh case 2), one generalised (score.test.js case 9), and net coverage rose because D3 and the degenerate-input block filled gaps the audit had identified but not costed. No (b) weak-signal case was touched: their concerns are valid and only their implementations are narrow, which the task explicitly excluded from deletion.

Recommendations (decision input, not applied here)

Feeding the sibling cleanup task:

1. .local_testing/ — the four smoke cases are subsumed by tests/e2e/cli-*.test.js and smoke-parity, and the harness around them does not work. Retire the directory, or fix H1–H7 as a set. A half-fixed Docker harness that nothing runs is worse than none: it reads as coverage. If retiring, test-base-commands.sh case 2's stale model ID (C) is the concrete evidence that nobody has run it in a long time.
2. score.test.js — add a projectForScoring case (D3); fix case 4 to pass an object with the key absent so the ?? null branch is actually covered (D1); drop the two redundant assertions in case 7 (D2). Keep case 9 despite its (d) classification — one line of test.
3. evals.test.js — keep as-is and keep it out of the default gate. Correct the README's fixture zone/archetype counts (D4). Consider a comment recording that POSIX-separator goldens make this the de-facto sv path-normalisation gate, so a future maintainer does not "simplify" it by normalising paths at comparison time.

Feeding the sibling cross-OS pipeline task:

4. smoke-parity itself is cheap and its two load-bearing cases (status-missing-rex, status-json) cover the real divergence surface. The question worth asking is not whether the comparison step earns its cost but whether cases #1–#6 earn theirs — they inflate the artifact and the sequence contract without adding parity signal.
5. If parity coverage is to be increased rather than trimmed, add cases that survive normalizeText: nested output paths, a command that prints a resolved absolute path, or a spawn-lifecycle case. Adding more static-text commands cannot improve the stage.

Raised by the cleanup task, not yet decided:

6. Harden the eval gate's separator detection. archetypeAccuracy returns 1.0 when the golden/actual path intersection is empty (see the correction in section B), so a Windows separator regression clears both archetype floors and is caught only by the zone scorer. Options: have archetypeAccuracy distinguish "no files to compare" from "perfect match" (a semantic change to a live gate — needs its own decision, since case A#3's intersection-denominator contract depends on the same guard); or assert a minimum compared- file count per fixture in evals.test.js, which leaves the scorer alone. The cleanup task only pinned and documented the current behaviour, deliberately — silently changing a gate's scoring semantics is not test cleanup.
7. Decide .local_testing/: retire or fix H1–H7 as a set. Recommendation 1 above still stands and was deliberately left unactioned. It is not test-case cleanup: README.md, CONTRIBUTING.md and packages/core/README.md all document this harness as the supported way to run the suite on Windows/Linux, and two completed PRD features (macos-local-docker-test-suite, local-test-suite-documentation-in-main) built it on purpose. Retiring it removes a documented developer workflow and touches three user-facing docs; fixing it means H1 (Linux entrypoint dead), H2 (--platform unparsed), H4 (Node 20 vs engines: >=22, which should fail --frozen-lockfile), plus H3/H5/H6 dead files and H7 doc drift — and Docker was never available to verify any base tag builds. Either way it is a decision for a human, and either way the three (b) cases in section C go with it.