Skip to the content.

Five upstream defects, pinned and re-proved

Seven capsules from two upstream Go repositories — four Kubernetes packages and one istio package — each found at the base commit of the pull request that fixed (or proposes to fix) it. Every capsule is content-addressed, repo.dirty: false, and verifyStatus: verified — the generated regression was run and was red on the unfixed tree.

Tool: v0.3.0-39-g142b237, go1.26.2, macOS 27.2 arm64.

Why these four Kubernetes cases exist upstream at all: pull-kubernetes-unit runs go test -count=1. Every one of them passes at -count=1 and fails only on a second run in the same binary, so the default CI order never sees them. That is the class this tool was built for, and it is the class its fixtures could previously only demonstrate on invented code — the three earlier real-world cases (slug, CLIProxyAPI, jamf-cli) are all order.

Method, identical for every case

git checkout <base commit of the fix PR>     # never the fixed tree
go test <plain command>                       # reproduce the report by hand first
go-flake-hunter repro . <Victim> --budget … --timeout … --capsule-dir … --format json
go-flake-hunter compile <capsule> --pkg . --verify   # must print: RED (verified)
git apply <upstream pull request>.patch
go test -tags goflakehunter -run '^TestFlakeRegression_<Victim>$' .   # must go green
go-flake-hunter replay <capsule> --pkg . -n 3      # must exit 3: never reproduced

The upstream patch is applied as published (gh’s .patch URL), never adapted. Where a patch touched a non-test file the tool did not need to know: the witness is built from the test bodies the repository already has.

The batch

Case Upstream Class Minimal witness Before After the patch
k8s client-go/rest k8s#143133 (open) repeat (-count) TestAuthPluginWrapTransport package red at -count=2, 4 tests green
k8s apiserver k8s#144048 (open) repeat TestObserveCompilation, TestRecordEtcdBookmark red at -count=2; a third package panics green
k8s scheduler/queue k8s#143132 (merged) repeat TestPriorityQueue_UpdateNominatedPodForNode red at -count=5 green
k8s scheduler/metrics k8s#142680 (merged) order TestQueuedEntitiesRecorder ← TestInFlightEventAsync + TestRecordGeneratedPlacements red at -shuffle=2 green — but one repeat witness survives it, see below
istio accesslog istio#39619, fix istio#39624 (merged 2022) order TestListenerAccessLog ← TestListenerAccessLogs red at -shuffle=2, -shuffle=3 green

Each case directory holds the same file set:

File What it is
capsules/gfh_*.json the capsule
*-repro.json / repro.json the repro run that produced it
*-compile.txt / compile.txt compile --verify, verbatim — the RED verdict
*regression_test.go.txt the generated test, as written
baseline-*.txt the failing go test on the unfixed tree, filtered to the assertions
upstream-fix.patch the upstream patch, as published
after-fix.txt the generated regression and the package, after the patch

Path substitution, and it is the only edit: the local checkout path became <repo> (so relDir reads <repo>/pkg/scheduler/metrics) and the capsule directory became <capsule-dir>. No field value, verdict or byte of output was otherwise touched — the verdicts are the ones the binary printed, which is why they are published as captured rather than re-rendered.

kubernetes client-go/rest

Base 7898e97f1724f72869caa6b354eda92845a6b7f8. Capsule gfh_f0726736eb90.

go test -count=2 ./rest
# FAIL TestAuthPluginWrapTransport / TestAuthPluginPersist
#      Test_WhenNilPersister_NoOpPersisterIsAssigned / TestThrottledLogger
#   "auth Provider Plugin \"pluginA\" was registered twice"
#   "expected 100 log messages, but got 2"

Two independent leaks in one package: auth plugins registered into a process-global map without cleanup, and lastLogTime left at fake-clock timestamps. One capsule names the first; the same -count=2 run names all four. The witness needs no polluter — repeat class, -count=10 in one binary, 2/2.

kubernetes apiserver

Base 7898e97f17. Capsules gfh_3874aa831864 (cel/metrics) and gfh_54a975bd9f97 (etcd3/metrics). One upstream PR, three packages, three different root causes.

Package -count=2 before Capsule
pkg/cel/metrics TestObserveCompilation, TestObserveEvaluation: unexpected count: 2 yes
pkg/storage/etcd3/metrics TestRecordEtcdBookmark: etcd_bookmark_total 2 vs 1 yes
pkg/storage/value TestLogTransformErr: panic: flag redefined: add_dir_header no — see below

The third package is a documented limit, not an omission. The defect is real and reproduces by hand (baseline-value-panic.txt), but go test -json never emits a terminal event for a test that panics the binary: the tool reports missing results for TestLogTransformErr: test run incomplete and exits 7. A repeat witness is a claim about a test outcome, and a dead process has none. after-fix.txt shows the same package green once the upstream patch is applied.

kubernetes scheduler/queue

Base 9b3380ed0f0882068abd7fb9bfbc816c595ae79e. Capsule gfh_6c64368b7e5a.

TestPriorityQueue_UpdateNominatedPodForNode deep-copies a package-level fixture, then clears NodeName on the original instead of the copy. Round 1 passes; every later round in the same binary sees an already-scheduled pod. Found by the repeat probe in two runs, no minimization needed, -count=10.

kubernetes scheduler/metrics

Base 3c690c189e5d367715dbd056a5c2009a476c8d77. Two capsules, and the second is the interesting one.

Order — gfh_7e7a0c4a10dc. Shuffled, TestQueuedEntitiesRecorder fails against a registry another test has already written. ddmin over the package reduced the preceding set to two tests:

TestInFlightEventAsync + TestRecordGeneratedPlacements → TestQueuedEntitiesRecorder

Repeat, and a leftover the merged fix does not close — gfh_b0ab94e0e871. Before the patch, TestNewMetricsAsyncRecorder also fails at -count=2 alone: it asserts on package-level counters it never resets. The upstream patch (metrics.go + profile_metrics_test.go) removes InitMetrics() from two other tests and fixes the shuffle symptom — the order witness goes green, -shuffle seeds 1/2/3 pass — but the repeat witness stays red after the patch:

git apply …/142680.patch
go test -count=2 -run '^TestNewMetricsAsyncRecorder$' ./pkg/scheduler/metrics
# --- FAIL: expected metric to have count 0, got 3

repeat-regression-after-fix is in after-fix.txt. This is the permanence claim working as intended: the committed regression outlives the patch and names what the patch left behind. Reported here as an observation about the pinned commit, not as a claim that upstream still carries it.

istio accesslog

Base ac641d08093c14ee53a71b7193a7e6a519db8159 (2022). Capsule gfh_1adc5e54f33f.

The upstream issue names -shuffle=on as the only trigger; on this hardware seed 1 passes and seeds 2 and 3 fail, so the report’s own reproducer was seed dependent. The witness is a sibling test whose name differs by one letter:

TestListenerAccessLogs → TestListenerAccessLog

TestListenerAccessLog resets the shared accessLogBuilder after it builds, so the first case in the table inherits whatever the previous test left; the fix moves the reset before the build. 2022 code needs one linker flag on a modern toolchain — the vendored golang.org/x/net reaches syscall.recvmsg by //go:linkname, which Go 1.23+ refuses:

GOFLAGS="-ldflags=-checklinkname=0" go test -shuffle=2 ./pilot/pkg/networking/core/v1alpha3

The flag is why the same package fails to build without it; it is not part of the flake. repro.json was produced with it set, and the capsule inherits nothing from it.

Blocked: cockroachdb/cockroach #160685

The sixth candidate is not in this batch. A plain checkout of 4c350c2bbe81c16574fb1beb8dd3171c84ad4afc does not compile ./pkg/cli:

pkg/server/diagnostics/diagnostics.go:17:2: no required module provides package
  github.com/cockroachdb/cockroach/pkg/server/diagnostics/diagnosticspb

Cockroach generates its protobuf packages at build time; they are not in the tree or in the module zip. Proving it needs ./dev generate go, which bootstraps its own toolchain. Recorded as blocked rather than claimed.

What this batch does not claim