A test that sometimes passes and sometimes fails is telling you something true.
When agents are listening to every signal, you can't afford to stop listening to that one.
An intermittent failure is always one of two real defects. Either the test has a race, an unsynchronised signal or a wall-clock dependency, or the product has a race, a missed signal or a leak under load.
Either way, it gets diagnosed and fixed before the change merges. No re-running the job, no skip, no .retry(), and no “unrelated” in the PR description.
“Re-run and hope is a coping strategy, not engineering.” From the SDKs' AGENTS.md.
In Agentic Loop Engineering, the test is the hard correctness condition that lets an agent climb to correctness. Zero-eyes approval trusts the loop's signals instead of a human read.
A noisy signal breaks both. Agents learn that red can be retried away. Real product defects, written off as “flakes”, ride through on the second run. Trust in every green drains away.
Reproduce it with loops, resource pressure or shorter timeouts. If you can't, reason about what differs between your machine and CI: load, filesystem, signal latency, parallel tests. These are the shapes named in our AGENTS.md files:
Retry back-off, backpressure, polling for eventual consistency and job timeouts are all about time. Tested against the real clock, they're slow and racy.
The SDKs route all timing through an injected clock. The Rust and Java SDKs go further and ban ambient time (Instant::now, sleep) outside the clock adapter, so a test clock really does control everything.
Determinism is a design property of the product, not just a test technique.
Every bug fix starts with a failing test that fails for the reason you expect. That proves the test can see this kind of bug. Then the minimal fix makes it pass.
The test is scoped to the class. If a generated method omits a required parameter, the test checks that all generated methods include their required parameters. Each bug fix then becomes a permanent guard against a whole category.
Illustrative method names.
The fix commit names the category (test defect or product defect), the signal the test used to rely on, and the deterministic signal it relies on now.
Where a generous timeout is still needed, a comment says it's a safety net, so no one tightens it back into a race.
An intermittent failure is a real defect, in the test or in the product. Diagnose it, control time, scope each fix to the class of defect, and write down which signal the test now trusts. That's what lets loops climb, and lets zero-eyes approval trust green.