The Verifier's Dividend

There is a number in METR's task-horizon data that gets far less attention than it deserves, and it isn't the doubling time.

METR's headline result is rightly famous: the length of task an AI agent can complete at a 50% success rate has been doubling fast — every seven months over the long run, and roughly every three months since 2024.¹ The less-quoted result sits one table away: the horizon at an 80% success rate is about five times shorter than the horizon at 50%. Same models, same tasks — just a different bar for "reliable."

That ratio is a measurement of something. This note works out what, and ends with an exchange rate we use constantly at the lab: what a good checker is worth, priced in the only currency this industry tracks — months of frontier progress.

Compounding is the whole story

Model an agent's task as a chain of steps — tool calls, edits, decisions. If each step goes right 99% of the time, a 200-step task finishes cleanly about 13% of the time. Not because anything dramatic happened — because 0.99 multiplied by itself 200 times is 0.13. Reliability compounds, and it compounds against you.

The arithmetic is brutal in a specific, useful way. Drag the slider:

Per-step reliability99.0%

Every curve ends the same way; the slider only decides how far out. Watch the two dashed markers — the point of 4-in-5 odds and the point where success is a coin flip. They stay ~3.1× apart no matter where you drag, which matters later. Ghost curves: 99% and 99.9%.

Two things worth noticing. At 99% per-step reliability — a number that would make any agent harness proud — success is a coin flip by step 69. And each additional "nine" of reliability multiplies the horizon by ten. Horizons are bought with nines, and nines of actor reliability are the most expensive commodity in this industry. You rent them from a frontier lab, at the frontier's price, on the frontier's schedule.

▸The survival math

With per-step reliability $p$, an $n$-step task succeeds with probability $P = p^{\,n}$, and the horizon — the task length at which success drops to a target $t$ — is $H_t(p) = \ln t / \ln p$. Near $p = 1$ this scales as $1/(1-p)$: one more nine, ten times the horizon. This is the constant-hazard reading of the METR curves, which Toby Ord spelled out shortly after the paper landed: an agent has a half-life, and survival decays exponentially with task length.²

There's a cheaper store.

Now add a checker

Put a verifier in the loop — type checks, property tests, sandboxed replay, a second model reading the diff. Now when a step goes wrong, one of two things happens: the checker catches it, and the step simply retries before the agent builds anything on top of the mistake; or the failure slips through, and the task is quietly poisoned.

That single change rewrites the compounding. Watch the gap open:

Actor reliability99.0%Verifier catch rate90%Retry budget2

Effective reliability99.899%

Horizon multiplier9.9×

Ceiling10.0×

Compute overhead+0.91%

Dashed curve: the bare actor. Solid curve: the same actor with a checker in the loop. Improve the actor and the multiplier climbs toward its ceiling — set by the checker alone. Notice what the overhead readout does while you drag: almost nothing.

The readouts are the story. A checker that catches 90% of failures stretches the agent's runway about tenfold. Push the actor's reliability up and the multiplier climbs toward a ceiling — a ceiling set entirely by the checker. And the compute overhead barely moves, because retries only cost money when something actually failed and got caught, which at these reliabilities is rare by construction.

Three things fall out of the model, and each one is load-bearing:

1. Checker quality converts one-for-one into runway. Catch 90% of failures, multiply the horizon by ten. Catch 99%, multiply it by a hundred. The same exchange rate as actor nines —
2. — except the price is different. At a 99% actor with a 90% checker, the whole scheme costs under 1% extra compute. Actor nines cost a training run. Checker nines cost an engineering sprint. Checking is cheaper than doing, and this is the machinery that monetizes the gap.
3. The multiplier doesn't care which model you run. It has no actor term in it. Whatever you deploy next year, the same checker still multiplies it.

▸Where the multiplier comes from

Let the checker catch a failed attempt with probability $c$, with a retry budget of $k$ per step, and let $q = (1-p)\,c$ be the probability an attempt fails and earns a retry. One geometric sum later, the effective per-step reliability is

$$p_{\text{eff}} = p \cdot \frac{1 - q^{\,k+1}}{1 - q},$$

and the dividend — the multiplier on horizon — is $D = \ln p / \ln p_{\text{eff}}$, independent of the success target. In the regime that matters ($p$ near 1, any retry budget at all), expand the logs and almost everything falls away: with $\varepsilon = 1-p$, $p_{\text{eff}} \approx 1 - \varepsilon(1-c)$, so

$$D \;\longrightarrow\; \frac{1}{1 - c}.$$

No $p$ anywhere — the ceiling belongs to the checker. The $k \ge 1$ condition is doing real work: with no retry budget, detection buys you loudness but no dividend. The expected overhead is $\mathbb{E}[\text{attempts}] - 1 \approx (1-p)\,c$ — at $p = 0.99$, $c = 0.9$, under 1%.

The exchange rate

Here is the framing that changed how we spend engineering time at the lab.

The industry already has a unit for "more horizon": waiting. METR's updated measurements put the frontier's 50% time horizon on an 89-day doubling since 2024.³ So when a checker multiplies your agent's runway tenfold, you can price it: ten times the horizon is a bit over three doublings, and three-and-a-third doublings at 89 days each is about ten months of frontier progress — delivered today, by your own harness, for around 1% overhead.

Verifier catch rate90%

Horizon multiplier9.9×how much longer your agent runs

Frontier equivalent9.7 mo3.3 capability doublings

Compute overhead+0.91%expected extra attempts

Fixed scenario: a 99%-per-step actor with two retries. The frontier's 50% time horizon has doubled roughly every 89 days since 2024 (METR, TH1.1), so a horizon multiplier converts into the months you would otherwise wait for models to catch up. The conversion is computed live from the same unit-tested model as every other instrument here.

That is the verifier's dividend, stated as a trade. When someone proposes spending a quarter chasing actor reliability, the first question is now: what's our catch rate, and what would raising it cost — in months?

What the 5× gap is telling us

Back to METR's ratio. The simple compounding model makes a firm prediction: the coin-flip horizon should sit about 3.1× beyond the 4-in-5 horizon. You can see it in Instrument 01 — the two dashed markers keep their spacing no matter where the slider sits. But METR measures roughly 5×, not 3.1×. The simple model is wrong in a directional way, and the direction is interesting.

A 5× gap means the hazard isn't constant — it falls as the task progresses. Agents that survive the opening act tend to keep surviving. Failures are front-loaded: the fatal move is the early wrong assumption that the next two hundred steps faithfully build on, not the thousandth tool call.

▸Why 5× implies front-loaded failures

Fit the simplest generalization — Weibull survival, $S(n) = \exp(-(n/\lambda)^\beta)$ — and the horizon ratio becomes $H_{50}/H_{80} = 3.106^{1/\beta}$. Setting that to the observed 5 gives $\beta \approx 0.70$. Shape below one means a falling hazard. We're fitting one parameter to one ratio — call it what it is, a back-of-the-envelope — but the sign of $\beta - 1$ is the robust part.

Which sharpens the engineering advice considerably: a uniform verifier is spending its catch-rate budget in the wrong place. Verify the plan like it's radioactive; spot-check the execution. The first ten steps of an agent run deserve a different class of scrutiny than the last hundred — the data, not just the intuition, says so.

A thousand agents walk into a task

Closed forms are tidy; production is a histogram. Instrument 04 runs the actual process — seeded, so you'll see what we see.

bare actorwith verifier (c=90%, k=2)

re-roll ↻

step 1where runs were poisonedstep 200

Completed132 / 1,000

Predicted13.4%

Failed silently868

Failed loudly0

Each run is a 200-step task at 99% per-step reliability (seeded, reproducible). The histogram shows where doomed runs were poisoned. Red is the dangerous mass — failures nobody noticed at the time. A bare actor fails silently by definition. Add the verifier and most of that mass simply disappears into retries; the red that remains is the 10% that slipped past the checker.

The toggle is the entire argument of this note, compressed. A bare actor doesn't just fail more — it fails silently, every single time, because nothing was watching. The verifier eliminates most of the failure mass and, just as valuable, converts the texture of what remains: caught-and-exhausted retries halt loudly where you can see them. The red that survives is exactly the share the checker missed — which is the number to attack next.

Silent failures are the expensive kind. They're the ones that ship.

What we do with this

Three working rules, all derived above, all in force at the lab:

• Buy nines from the verifier store first. Tenfold runway for one percent overhead is the best trade in agent engineering. Exhaust it before paying actor prices.
• Front-load the scrutiny. The plan review is worth more than the step review — the failure data says so, not just the intuition.
• Engineer failures to be loud. The verifier's second product, after the dividend, is the conversion of silent failures into loud ones. Loud failures cost a retry; silent ones cost a postmortem.

And one falsifiable claim, so this note earns the name field note rather than opinion: the model says the gap between the coin-flip horizon and the 4-in-5 horizon is set by the harness, not the model — so as models improve, it should hold near 5× rather than closing toward 3.1×, unless harnesses change. That's checkable against every future METR release. If it breaks, our constant-catch-rate assumption breaks with it, and we'll write the follow-up.⁴

This model is also becoming an instrument in the literal sense: Mirrormere, our verification harness, measures the catch rate a team actually has and prices what raising it is worth — in months.

The instruments on this page run the same math we unit-test in the site's repository — the essay and the code can't drift apart without CI noticing.

Footnotes

1. Kwa et al., Measuring AI Ability to Complete Long Tasks (METR, 2025). The 50%-horizon doubling time is ~7 months over 2019–2025, faster in the most recent span; the 80% horizon runs roughly 5× shorter than the 50% horizon across models. ↩

2. Toby Ord, The Half-Life of AI Agents (2025) — the observation that METR's curves are consistent with a constant per-minute hazard rate, i.e. exponential survival. ↩

3. METR, Time Horizon 1.1 (January 2026). The expanded 228-task suite puts the post-2024 doubling time at ~89 days. The exchange-rate arithmetic: a multiplier of $D$ is worth $\log_2 D$ doublings, each doubling ~89 days of trend. ↩

4. Known liberties: steps aren't independent, verifiers have false-positive rates (which tax throughput, not correctness), and a verifier whose blind spots correlate with the actor's failure modes has a lower effective catch rate than its benchmark suggests — the case where "use a second model as the checker" quietly underdelivers. Each of these moves the constants, not the shape. ↩