jamf-cli: an order-dependent test, shrunk to one polluter
Repo: Jamf-Concepts/jamf-cli at
977b704, 2026-11-11. Issue
#370, open since
2026-09-07. The fix merged on November 12, 2026 in
#474
(62feabf4168c816114218b94f932e9aecac249aa).
Attribution. The maintainer (ktn-jamf) wrote the issue and named the
mechanism: the package keeps cobra’s parsed state in package-level vars, so any
test that drives the root command leaves it set. The contribution here is the
minimal polluter, the deterministic witness, and the fix — not the diagnosis.
Why this repository
Not a seeded exercise: a real product, pushed the same day the hunt ran, with an admitted order dependence that had been open for two months. The package has 1736 tests and a full run costs 60-200 s, which is why nobody had shrunk it by hand.
Reproduction at the base commit
go test ./internal/commands/ -count=1 # ok, 61s
go test ./internal/commands/ -count=1 -shuffle=10 # FAIL: the victim
go test ./internal/commands/ -count=1 -shuffle=1234 # FAIL: the victim
The seeds named in #370 are partly stale: -shuffle=2 and -shuffle=7 pass on
977b704, so the failure reproduces on 10 and 1234.
Finding the polluter
repro --from-json on the failing -shuffle=10 log (38 MB; the victim ran at
index 1873 of 3146) shrank 1873 preceding tests to one:
flake found: TestCheckAPIMatchRefusesAPlatformCommandOnAnInstanceProfile
package : github.com/Jamf-Concepts/jamf-cli/internal/commands
class : order
polluter : TestResolveAuth_URLNormalization
run : 3/3 failed
Wilson 95% : 43.9%–100.0%
deterministic: yes
capsule : gfh_e628d198b8b2
status : confirmed
Files in this directory:
| File | What it is |
|---|---|
capsules/gfh_e628d198b8b2.json |
the capsule; repo.dirty is false, base commit 977b704 |
repro.txt |
the repro --from-json run that produced it (the checkout’s absolute path is replaced by <repo>; nothing else is edited) |
shuffle-before.txt |
seed sweep on pristine 977b704 |
shuffle-after.txt |
the same sweep with the patch |
shuffle-seed7-alone.txt |
the A/B that ruled out a load-induced failure |
compile --verify wrote the regression file and reported
verification: RED (verified — the flake is still there).
Mechanism
credentialSource (internal/commands/gateway_coverage.go:171) builds the
sentence the victim asserts on from four package-level vars: token,
tokenFile, clientID, clientSecret.
resolveAuth (internal/commands/root.go:703-734) writes them: it caches
JAMF_TOKEN, JAMF_CLIENT_ID, JAMF_CLIENT_SECRET and token-file contents into
the globals. Every test that calls resolveAuth therefore leaves them set.
- Polluter
TestResolveAuth_URLNormalizationresolves a profile whose token isenv:TEST_URL_NORM_TOKEN, which setstoken. - Victim asserts the refusal names
profile "pro-instance"; withtokenset it readsthe JAMF_TOKEN environment variableinstead:
gateway_coverage_test.go:95: the refusal does not mention "profile \"pro-instance\"":
... (auth-method oauth2, from the JAMF_TOKEN environment variable)
tokenFile leaks the same way through TestResolveAuth_TokenFromFile
(the token in <path>), and so do clientID/clientSecret.
In source order the two assertions run before any polluter
(gateway_coverage_test.go sorts before pro_* and root_test.go), which is
why the default suite is green and CI never shuffled.
Fix
isolateAuthGlobals(t) already exists in platform_scope_flags_test.go — added
in #410 for this exact
failure mode. The two tests that assert on credentialSource output now call
it. 11 lines, one test file, no production change.
Verified
| Run | before (977b704) |
with the patch |
|---|---|---|
-count=1 |
ok | ok |
-shuffle=2 |
ok | ok |
-shuffle=7 |
ok | ok |
-shuffle=10 |
FAIL | ok |
-shuffle=1234 |
FAIL | ok |
A witness built from the capsule (polluter → victim, build-tagged go-flake-hunter)
is RED before and GREEN after.
One measurement lesson: an early “seed 7 fails” reading was taken while a second
full suite ran on the same machine and did not reproduce when run alone — see
shuffle-seed7-alone.txt. A shuffled run is deterministic in order but not in
wall clock, and this package has tests sensitive to the latter.
Still order-dependent (measured, untouched)
| Seed | Failing test |
|---|---|
-shuffle=99991 |
TestProGroupAction_UniqueGroupNameSendsOneWriteThroughTheRoot |
-shuffle=on (random) |
TestDryRun_ProUploadIsNeverSent |
Each needs its own polluter before a fix can be trusted, so they are left out of the PR.
Re-measured on 2026-11-17 at the merged commit 62feabf: seed 99991 no longer
fails in TestProGroupAction_UniqueGroupNameSendsOneWriteThroughTheRoot — the
same seed now fails in TestLogflushPolicy_SendsOneDeleteAfterLookup. The patch
isolated two leaking tests; the shuffle moved on to the next one.