The Software GardenPoint them at a number ← Contents
The Software Garden · Quality · 08

Point them at a number

A test tells an agent pass or fail. A measurement tells it how far it has to go. Give an agent a target, a load generator and the system’s telemetry, and it can find and remove its own bottlenecks.

scroll ↓
Telemetry is a test

The same loop, with a measurement as the check.

In Agentic Loop Engineering, the agent climbs towards green tests. Here it climbs towards a number: throughput under load, tail latency, memory at peak.

Each turn is longer than a unit test run: write, build, deploy, put it under load, read the telemetry. The agent then forms a hypothesis about the bottleneck and the next turn tests it.

A pass/fail check can only say “not yet”. A number shows which way to go.

experiment1ceiling: measuring…
Worked example · the Nano BPM engine

Most hypotheses were wrong. That’s what the loop is for.

In July 2026, agents hunted the throughput ceiling of a three-node Nano BPM cluster on Google Cloud. Every step below is from the commit history. Step through the notebook.

Refuted 0Fixed 0Claims corrected 0
Measurements lie too

If the loop trusts a number, check the instrument.

Two of the turns in that notebook weren’t about the engine at all. One “pinned” thread was an artifact of how top reports a parked thread. And every cluster benchmark had run with replication switched off, because a script set the replication factor but never enabled Raft.

Both were caught the same way: a controlled A/B that disagreed with the story. A mis-wired measurement is a flaky test with better formatting. The loop is only as good as its instruments, so the instruments get tested too.

“Throughput numbers stand; replication attributions do not.”From the correction commit, which annotated every affected passage in the performance docs.
“The earlier ‘100%-pinned main thread’ was a top -H artifact.”From the A/B that found the new dispatcher made no difference.
Worked example · the SDKs

Map the performance envelope, then build the guidance from it.

For the SDKs, a performance matrix runs workers for eight SDKs in seven languages across worker counts, process topologies, transports, handler types and cluster sizes: 1,280 scenarios. It also sweeps individual settings, so defaults are measured, not guessed. For what this changed in practice, see the case study.

28k → 30kprocess instances per second on one node: 7% more load, half the throughput, p99 from about 191 ms to about 14 s. The knee is a cliff.
157 → 7,458/sinstance creation with 1 against 64 producer connections. Creates queue behind each other on one connection, so connections are the lever.
146 vs 117/sa C# worker polling every 25 ms against the old default, measured on Google Cloud. The new default shipped in the harness, with the SDK’s retry gaps written up.
The oracle doesn’t have to be a test you wrote

Seven SDKs can check each other.

A conformance harness generates a call for every operation in the API spec, sends it through all seven SDKs, and captures the HTTP request each one would send. Nobody writes the expected output: the SDKs are compared with each other.

Where they disagree, at least one of them is wrong, and whoever owns it gets a precise defect to fix. The signal comes from the system, not from a person predicting it.

Telemetry under load“How fast, and where does it break?”
Cross-implementation comparison“Do they all mean the same thing?”
The spec“Is that what the API says?” (derivation over duplication)
The next step

Point the same loop at production.

Everything above runs against a load generator. The next step is to use the telemetry of the deployed system itself: real traffic, real payloads, real failure modes that no stress test thought to try.

It is the same signal from further out, and nothing about delivery has to change. The agent reads production, forms a hypothesis, tests it in the stress loop, and its change goes back through the usual loops and the merge gate like any other.

What changes

Told what to optimise, agents start the work themselves.

That moves part of the vision edge. No one predicted that the ceiling was a single exporter thread. A person set the goal; the loop found the work. The human part of the vision becomes choosing what to optimise, and what must not be traded for it.

Humans set

  • The objective: “maximum throughput at p99 under 200 ms”, not a backlog of tasks.
  • The guardrails: numbers that must not get worse. In the ceiling hunt, durability was one: async writes were measured, and stayed off.
  • The budget and the stop: how much compute, and when gains are too small to matter.
  • The ranking: which objective wins when two pull against each other.

Agents do

  • Read the telemetry and pick the next bottleneck.
  • Form a hypothesis and design the experiment.
  • Make the change, run it under load, keep or discard it.
  • Write it down: what was tried, what was measured, what was wrong.

Optimise a single number and an agent will find ways to move it that you didn’t intend (Goodhart’s law). The guardrails are what keep “faster” meaning faster.

So

A number is a test that points somewhere.

Put agents in a loop with load and telemetry and they will hunt bottlenecks, disprove their own theories and write down what they learned. Check the instruments as carefully as the code. Then give them an objective and guardrails, and let them find the work.