Real-world witness validation
Validated on November 10-11, 2026, macOS ARM64, Go 1.27.3, against three real repositories with upstream test-isolation reports. All checkouts were clean when searched. No test body was modified to make discovery succeed.
| Case | Base commit | Upstream fix or proposal | Replay before | Compile before | Generated test after fix | Replay after |
|---|---|---|---|---|---|---|
| slug | 1eab3167af84fc96dbabe7b5d9c22d6d4c214800 |
gosimple/slug#83, 76470d7e1c9c8bb5cf9b9df392b9fb5f35b77e9a |
5/5 failed | verified RED | PASS | 0/2, exit 3 |
| cliproxy | 0074897481fee1dcd595c1caa1d57e6a379b6be6 |
CLIProxyAPI#7356, e056459bd00de0c63bbf5e09879704b0d53fd142 |
5/5 failed | verified RED | PASS | 0/2, exit 3 |
| jamf-cli | 977b704 |
Jamf-Concepts/jamf-cli#474 (merged November 12, 2026) | 3/3 failed, deterministic | verified RED | PASS | default suite + -shuffle 2/7/10/1234 green |
These targeted historical cases demonstrate reproduction and permanence. They
do not estimate recall or false-positive rate over arbitrary repositories.
Only the targeted generated regression is claimed green after each fix; no
claim is made that the entire upstream suite is green. gosimple/slug#83 is
merged (2024); CLIProxyAPI#7356 is an open proposal, applied locally from its
patch excerpt to prove the regression turns green.
The recordings in this directory (repro.json, capsules/, compile.txt,
report.md, regression_test.go.txt) are the verbatim output of the release
that produced them, so they still carry that release’s capsule prefix and build
tag. They are evidence, not documentation: rewriting them to match a later name
would make them no longer evidence.
The jamf-cli case is the only one where the tool’s output is the contribution
rather than a re-validation: the flake was already reported by a maintainer, and
repro --from-json turned 1873 preceding tests into the single polluter the fix
needs. Attribution is recorded in that case’s README — the diagnosis is the
maintainer’s, the minimal witness and the patch are not.
Second wave: five defects at fixed commits (November 15, 2026)
Validated on November 15, 2026, macOS ARM64, Go 1.26.2,
v0.3.0-39-g142b237 — seven capsules from four Kubernetes packages and one
istio package, every one pinned at the base commit of its fix pull request.
Four of the five are repeat class, which the three cases above do not cover:
upstream Kubernetes CI runs -count=1, so those defects are invisible to it.
| Case | Upstream | Class | Minimal witness | After the patch |
|---|---|---|---|---|
| k8s client-go/rest | #143133 (open) | repeat | TestAuthPluginWrapTransport |
package green at -count=2 |
| k8s apiserver | #144048 (open) | repeat | TestObserveCompilation, TestRecordEtcdBookmark |
green; a third package panics — reported, not claimed |
| k8s scheduler/queue | #143132 (merged) | repeat | TestPriorityQueue_UpdateNominatedPodForNode |
green at -count=5 |
| k8s scheduler/metrics | #142680 (merged) | order | TestInFlightEventAsync + TestRecordGeneratedPlacements → TestQueuedEntitiesRecorder |
green — one repeat witness survives the patch |
| istio accesslog | #39619 → fix #39624 (merged 2022) | order | TestListenerAccessLogs → TestListenerAccessLog |
green at -shuffle 2/3/7 |
The scheduler/metrics case is the one worth reading twice: the merged patch
removes the shuffle symptom, the order regression turns green, and a second
witness found in the same file (repeat, TestNewMetricsAsyncRecorder) stays red
on the patched tree. That is the permanence claim doing its job — the committed
regression outlives the fix and names what the fix left behind.
A sixth candidate (cockroachdb/cockroach #160685)
is not in the batch: a plain checkout of that commit does not build ./pkg/cli
because Cockroach generates its protobuf packages at build time. Blocked, and
recorded as blocked.
Full narrative, every command and every raw file: upstream-2026-11.
Repeat the validation
From the go-flake-hunter checkout:
scripts/verify-real-world.sh /tmp/go-flake-hunter-real-world-evidence
The script builds go-flake-hunter from the current source, creates disposable checkouts, downloads each repository’s pinned dependencies, records discovery, replays five times, generates and verifies the regression, applies the exact upstream test-file patch, and runs the same generated regression again. It also requires replay to return exit 3 after the fix. Network access and support for local HTTP test listeners are required. Output files are retained; disposable checkouts are removed.
Each case directory contains the confirmed capsule with verifyStatus, the
search report, replay outputs, compile verification output, generated regression
source (as text), the upstream patch and the green result. Upstream patches are
small excerpts from the linked projects; their source remains under the
slug MPL 2.0 license and
CLIProxyAPI MIT license.
slug: custom substitutions leak into a default slug test
Package: repository root. Minimal witness:
TestSlugMakeUserSubstituteLang → TestSlugMake
The custom substitution test leaves its CustomSub map active. A later default
slug conversion observes the custom replacement. Upstream #83 resets the
custom substitution maps; applying that patch makes this generated regression
pass.
Discovery command, from the pinned upstream checkout:
GOFLAGS=-shuffle=3 go-flake-hunter repro . TestSlugMake --budget 200 --timeout 5m \
--capsule-dir /tmp/slug-capsules --format json
The seed exposes the reported custom-substitution relationship before the package’s separate truncation-state leak can obscure it. The seed is only a search aid: replay and the generated regression run without it, using the explicit polluter→victim chain.
CLIProxyAPI: a websocket client leaks into the model catalog
Package: sdk/api/handlers/openai. Minimal witness:
TestResponsesWebsocketHomeSelectedAuthCallbackPinsAndReusesFirstSelection
→ TestCodexClientModelsApplyPatchRouting
The websocket test registers a process-global model-registry client and leaves
it present. The catalog test later sees an unexpected model/capability entry.
Upstream #7356 adds t.Cleanup to unregister that client. The same generated
regression passes with that patch applied.
Discovery command, from the pinned package directory:
GOFLAGS='-run=^(TestCodexClientModelsApplyPatchRouting|TestResponsesWebsocketHomeSelectedAuthCallbackPinsAndReusesFirstSelection)$' \
go-flake-hunter repro . TestCodexClientModelsApplyPatchRouting --budget 200 --timeout 5m \
--capsule-dir /tmp/cliproxy-capsules --format json
The package sweep is scoped to the two tests named in the upstream report;
isolated and witness runs supply their own -run, overriding that default.
Replay and generated-test verification need no search filter. An unrestricted
package search also found the same pair during investigation, but the repeatable
script uses the narrower scope to avoid unrelated slow/timing-sensitive tests.
Local integration reductions
testdata/fixtures/substitution-leak and testdata/fixtures/registry-leak are
original stdlib-only models of these mechanisms. They are exercised by
TestCompileVerifyReproducesEachClass. These reductions are not counted as
real-world findings; the pinned upstream checkouts above supply that evidence.