The Software GardenAll software fails somewhere ← Contents
The Software Garden

All software fails somewhere

Keep the defect rate falling as throughput rises.

Every system fails in some way, under some condition. The question isn't how to reach zero defects. It's how to keep the defect rate flat, or falling, while throughput goes up. Going slow isn't the safe option: it's the other way to lose.

Scroll ↓
First, what isn't a trade-off

The signal is non-negotiable. Everything else is engineering.

There are no flaky tests isn't a preference we balance against speed. Every loop climbs on its checks, so a check that lies breaks everything built on it.

But a perfect signal doesn't mean a perfect product. Tests only check the conditions someone thought of. The product will still fail under some conditions nobody has tested yet. Where we look for those failures, how hard, and how early: those are the trade-offs.

🔒 Not negotiable

  • Every red means something
  • Required checks that can actually fail
  • A baseline that starts green
  • Escaped defects get a root cause

⚖️ Trade-offs

  • Which checks run in the inner loop, and which later
  • How much scenario coverage before merge vs. after
  • Which defect classes to design out first
  • How much residual risk a change can carry
An honest baseline

If eyeballing the code at human speed were perfect, we'd never have had bugs.

We had bugs. Every team, every decade, with careful engineers reading every line and approving every PR.

So the comparison isn't “agents vs. perfection”. It's agents vs. a process that already had a defect rate, and a throughput capped by how fast humans can read.

Human review catches real things. It just doesn't scale, and it was never a guarantee.

WaterfallMonths of design, formal reviews, QA phases. 🐞🐞🐞
AgileSmall batches, pair programming, code review. 🐞🐞🐞
DevOpsCI/CD, every PR approved by a human. 🐞🐞🐞
AgenticThe question is the same one we always had: at what rate, and how fast do we learn from them?
The goal

Scale throughput. Hold the relative defect rate. Then lower it.

At ten times the changes, ten times the absolute defects at the same rate is the honest starting point. The goal is to keep defects per change flat, and ideally make it fall.

What you can't do is reach that goal with a method that limits velocity. A human approval gate holds the rate by capping throughput, so the extra agent capacity just turns into a queue.

Agent throughput1×
Changes shipped / wk100
Defects / 100 changes3.0
Defects / wk3
Shipped vs. produced
Waiting for review
 

Illustrative model, not measured data.

Move the reasoning

Don't reason about the code at human speed. Reason at compiler speed.

A human reading a diff checks one change, once, slowly. A type system or an architecture rule checks every change, every time, in seconds, and the agent learns about the failure before it has finished the task.

Encode the reasoning once as structure: types that make illegal states unrepresentable, module boundaries enforced in the build, rules that enforce themselves. A whole class of defect stops being something anyone has to notice.

👀 Human review
minutes–days / PR
Good at intent and taste. Tires, skims, and caps throughput.
🤖 Agent review loop
minutes / PR
Scales with the fleet, but it's probabilistic: a strong signal, not a proof.
🧱 Architecture rules
seconds / build
Every import, every boundary, on every change.
⚙️ Type checker
milliseconds / edit
Deterministic. Unrepresentable states can't be committed by anyone.
Then expose what's left

Every layer has holes. Stack them so the holes don't line up.

Types and rules remove whole classes of defect, but not all of them. Behaviour that only appears when components meet, under load, across versions or after a restart is invisible to the compiler.

So we scaffold defense in depth: unit tests, integration tests and scenario tests on top of the types and rules, each catching what the layers before it let through. No single layer has to be perfect. The stack does.

 

Illustrative catch rates.

Aim at the product

Don't ask “is the code right?” Ask “under what conditions does the product fail?”

Integration and scenario tests aren't there to re-check the code line by line. Their job is to find the conditions the product breaks under, deliberately, before a user does.

Agents make these suites cheap to build and to grow. That's where the extra capacity should go: not into more code that nobody tests, but into more ways of trying to break it.

🔁 Restart mid-flightKill a node halfway through a process. Does it resume exactly once?
🌐 Partition and latencyDrop and delay traffic between components. Do timeouts and retries hold?
🕰️ TimeClock skew, DST, timers firing late. Does anything assume wall-clock order?
📈 Load and backpressureTen times the traffic. Does it slow down gracefully or fall over?
🔀 Version skewOld client, new server; rolling upgrade halfway done.
🔑 Permissions and tenancyThe wrong user, the wrong tenant, an expired token.
Close the loop

Every failure the outer layers find hardens the inner loops.

A scenario test that finds a failure has done half its job. The other half is asking: what's the earliest, fastest layer that could have caught this class of defect? Then move it there.

Over time, defect classes migrate inward: from production to scenario tests, from scenario tests to unit tests, from tests into the type system. The outer layers keep exploring new failure conditions, and the inner loops get faster and stricter. That's how the rate goes down while throughput goes up. (More on this in Agentic Loop Engineering.)

1
A scenario test failsRestart mid-process creates a duplicate job.
2
Fix the instanceRed, then green, in the scenario suite.
3
Find the classAny handler that isn't idempotent on redelivery.
4
Move it inwardA unit-level redelivery harness for every handler, and a type that won't build a handler without an idempotency key.
↻ the scenario suite goes looking for the next condition
So

Accept that it fails. Engineer where you find out.

Human speed was never defect-free, and it can't scale. Keep the signal honest, move reasoning to compiler speed, stack integration and scenario tests to expose the conditions the product fails under, and feed every finding back into the inner loops. Then throughput goes up and the defect rate goes down.