The 2026 Flaky-Tax Index

The dollars and engineer-hours a team loses every month to flaky CI failures, indexed by team size, and why the human cost dwarfs the compute cost.

Executive summary

Flaky tests are the cost that never appears on an invoice. A build goes red, an engineer retries it, it goes green, and everyone moves on, but the wasted compute and the broken concentration are real and they recur many times a day. This index puts a number on that tax: the modeled monthly dollars and engineer-hours a team loses to failures that were never genuine, sized by team. The goal is to make a cost that is normally invisible large enough to act on, because a cost nobody measures is a cost nobody fixes.

The compute side is straightforward to price: re-run minutes at the published per-minute rate. Every spurious red build that gets retried spends real minutes recomputing a result that was already correct, and those minutes show up on the bill even though they bought nothing. The human side is larger and slower to see, because every spurious red build pulls an engineer out of flow to investigate something that was not broken. Our model puts the monthly Flaky-Tax Index for a 20-engineer team at $1,840 and 46 lost engineer-hours, and both numbers grow super-linearly as merge traffic rises with headcount.

The reason the index grows faster than the team is that flakiness is multiplicative with activity. A larger team merges more often, runs more branches, and triggers more pipeline runs, and every one of those runs carries its own independent chance of a flaky failure. Double the team and you more than double the runs, the branches, and the merge traffic, so the flaky tax climbs faster than headcount rather than tracking it linearly. That is what turns flakiness from a small-team nuisance into a structural cost for a growing organization.

Crucially, most flaky failures are transient and mechanical: network blips, registry timeouts, out-of-memory kills, shared-state ordering. They pass on a clean retry, which is exactly why self-healing runners remove the tax without anyone touching test code. The build was never really broken; the environment hiccuped. Quarantining or rewriting the test is the wrong first response for the bulk of flakes, because the test is fine. What failed was the infrastructure around it, and the fix is to retry on a fresh environment automatically.

The managed line in this index captures that. It retries transient failures automatically, before a human is pulled in, and at a per-minute rate about 70% below the hosted blend, so it removes both the human cost of the context switch and most of the residual compute cost. The difference between a team that manually retries flakes and a team whose runners self-heal is not a small efficiency, it is the gap between paying the full flaky tax and paying a fraction of it, which this report quantifies tier by tier.

The flaky tax grows faster than headcount

Doubling team size more than doubles the flaky tax because merge traffic, branch count, and pipeline runs all rise together, and each carries its own flake probability. In our model the index climbs from a few hundred dollars at five engineers to tens of thousands at two hundred, a super-linear curve that makes flakiness a structural problem for growing orgs rather than a nuisance for small ones.

The mechanism is straightforward once you see it. The flaky tax is roughly the number of pipeline runs times the per-run flake probability times the cost of each flake. Headcount drives run volume more than linearly, because more engineers means more branches and more frequent merges per engineer, not just more engineers. So the run-volume term grows super-linearly, and the tax grows with it. A larger team is not just paying the small-team tax more times; it is paying a tax that compounds against its own growth.

This is why flakiness tends to arrive as a surprise at scale. A team that barely noticed flaky tests at five engineers crosses a size where the run volume is high enough that the same per-run flake rate produces a constant background hum of spurious failures. Nothing about the tests changed; the activity around them did. The chart below shows the curve, and its shape is the argument for treating flakiness as a first-class reliability concern before the team gets large rather than after.

• The tax is run volume times per-run flake probability times cost per flake.
• Run volume grows super-linearly with headcount via more branches and more frequent merges.
• The same per-run flake rate produces far more absolute pain at scale, so the curve steepens.

The human cost outweighs the compute cost

Wasted re-run minutes are real money, but the larger share of the tax is the engineer-hours lost to investigating red builds that were never broken. A 20-engineer team loses dozens of hours a month to context-switching alone, time that does not appear on any bill and is rarely attributed to flakiness when it is spent. The minutes are visible on the invoice; the hours are buried in slipped estimates and interrupted focus.

The context switch is the expensive part, not the retry click. When a green change comes back red for no reason, the engineer does not simply press retry and move on. They stop what they were doing, read the failure, try to reproduce it, suspect their own change, and only then conclude it was a flake and rerun it. That whole loop is destroyed concentration, and the cost of breaking flow on focused work is far higher than the wall-clock minutes the loop consumes.

This is why the engineer-hours chart climbs even more steeply than the dollar chart. At two hundred engineers the model puts the lost hours in the hundreds per month, a figure equivalent to multiple full-time engineers spent doing nothing but investigating failures that were not real. No team would knowingly staff that, yet flakiness staffs it silently, which is the strongest argument for removing the failure before a human ever sees it rather than making the human faster at dismissing it.

• The dominant cost is broken focus, not the minutes a retry consumes.
• The investigate-reproduce-suspect-rerun loop destroys concentration on the engineer's real work.
• At large headcount the lost hours equal multiple full-time engineers doing nothing but dismissing false alarms.

Most flakes are transient and pass on a clean retry

The dominant root causes are mechanical: network and registry timeouts lead, test ordering and shared state follow, resource limits such as out-of-memory and disk pressure come next, and genuine race conditions in the tests themselves are the minority. Three of those four clear on a fresh attempt because nothing in the test logic was wrong; the environment around it hiccuped and then recovered.

This distribution is the central fact of the report, because it determines the correct response. If most flakes were genuine nondeterminism in the test code, the fix would be to rewrite or quarantine tests, slow and manual work that competes with shipping. But the chart below shows that the genuine-race category is the smallest slice. The bulk is infrastructure: a registry that timed out, a runner that ran out of memory, a previous test that left shared state behind. Those are not test bugs.

Because the majority is mechanical, automated retry of transient failures recovers most of the tax without changing application or test code. The team does not need to audit its test suite to remove the dominant share of its flaky tax; it needs the platform to recognize a transient signal and rerun on a fresh environment. The minority of genuine races still deserves test-level attention, but it is a backlog item, not the structural cost, and treating the whole problem as a test-quality problem misdiagnoses where the money goes.

Self-healing removes the tax that retries only hide

Manually retrying a flaky build still spends the engineer-hour even if the second run is green. The retry click is cheap; the context switch that preceded it is not. So a manual-retry workflow does almost nothing about the largest component of the flaky tax, the human one, because the human has already been pulled in by the time they reach for the retry button.

Automated self-healing on the runner detects the transient failure and re-runs it before a human is pulled in, which is where the real saving comes from. The chart below walks the same fifty-engineer team through four configurations: manual retry on hosted runners is the most expensive, automated retry helps, self-healing on hosted runners helps more, and self-healing on managed runners is the cheapest. The big step down is the move from manual to automated, because that is the step that protects the engineer's focus.

On managed runners the retry also costs about 70% less per minute, so the residual compute tax shrinks on top of the human saving. The two effects stack: self-healing removes the context switch, and the managed rate makes the recovery minutes themselves cheap. That is why the self-healing-managed bar sits well below the others, and why the gap between it and the manual-retry bar is the real size of the flaky tax a team can remove.

• Manual retry leaves the largest cost, the context switch, fully in place.
• The biggest saving is the move from manual to automated retry, which protects engineer focus.
• Managed runners then price the recovery minutes about 70% below the hosted blend, shrinking the residual compute tax.

Flakiness is a reliability metric, not a test-quality footnote

Because flaky failures slow delivery and erode trust in CI, they map directly onto the DORA reliability frontier. Elite teams keep change-fail rate in the 0-15% band and recover in less than one hour, and a pipeline that fails for non-code reasons works against both, because every spurious red build muddies the signal that those metrics depend on. A team cannot reason cleanly about its real change-failure rate when a meaningful share of its failures are noise.

Treating the flaky tax as a first-class reliability number reframes it from a backlog chore into a delivery-performance lever. With CI adoption near 76% of professional developers, almost every team is exposed to this, and the ones that measure their flaky tax can defend the time spent removing it the way they defend any other reliability investment. The ones that leave it unmeasured keep paying it as an untracked drag on everything else.

The deeper point is that flakiness is where reliability and cost meet. A flaky pipeline is both more expensive, through wasted minutes and lost hours, and less reliable, through eroded trust and muddied metrics. Fixing it improves both at once, which is rare. Most optimizations trade one axis for another; removing the flaky tax improves the bill, the delivery metrics, and the engineering experience simultaneously, which is why it earns a place near the top of any reliability roadmap.

A green-after-retry pipeline is hiding a number you should track

Many teams believe they do not have a flakiness problem because their builds eventually pass. They do pass, after a retry, which is exactly the condition that hides the tax. The pipeline is green, the change merges, and the cost is paid silently in minutes and focus rather than surfaced as a failure anyone owns. A pipeline that is reliably green only on the second attempt is not reliable, it is expensive.

The metric to track is the retry rate, or equivalently the share of builds that fail at least once before passing. It is cheap to instrument and it directly exposes the flaky tax that a pure pass-rate metric conceals. A team watching only its eventual pass rate sees a healthy green dashboard; a team watching its first-attempt pass rate sees the real reliability of its pipeline and the size of the tax it is paying to keep the dashboard green.

Once the number is visible it becomes defensible to act on. The retry rate ties directly to the dollar and hour figures in this index, so a team can translate a high retry rate into the modeled monthly cost and justify the work to remove it. Without the number, flakiness stays in the category of things everyone grumbles about and nobody is funded to fix, which is precisely why it persists.

Compute and human cost are best removed by the same mechanism

The two halves of the flaky tax, the wasted minutes and the lost hours, are often treated as separate problems, but a single mechanism removes both. Self-healing on the runner reruns the transient failure on a fresh environment automatically, which means the engineer never gets pulled in, removing the human cost, and the rerun happens on a managed runner at a lower per-minute rate, shrinking the compute cost. One change addresses both axes.

This is more efficient than the common alternative of attacking the two halves separately. A team might tune retries to save compute and run flakiness-reduction sprints to save engineer time, two efforts that compete for the same scarce attention. Routing transient failures through automated recovery on a cheaper runner collapses both efforts into one piece of infrastructure that pays back continuously rather than as a series of one-off projects.

The residual genuine-race minority remains, and it is correctly handled at the test level, but it is the smallest slice on the root-cause chart. For the mechanical majority, which is where the money and the hours actually go, the right tool is a runner that heals itself. Sizing your own flaky tax against this index is the first step; routing it through self-healing managed runners is how the index comes down.

Recommendations

Instrument first-attempt pass rate, not just eventual pass rate

A pipeline that is green only after a retry is hiding its flaky tax. Track the share of builds that fail at least once before passing; it is cheap to measure and it exposes the cost that a pure pass-rate dashboard conceals, which is the prerequisite for funding the fix.

Size your own tax against the index before prioritizing

Use your retry rate, run volume, and team size to place your team on the dollar and hour curves in this report. Translating a vague sense of flakiness into a modeled monthly figure is what turns it from a grumble into a defensible reliability investment.

Auto-heal transient failures rather than quarantining tests

The dominant root causes are network timeouts, resource limits, and shared-state ordering, all of which clear on a fresh attempt. Retry transient failures automatically on a fresh environment to remove the mechanical majority of the tax without touching test code, and reserve test-level work for the genuine-race minority.

Remove the failure before a human sees it, not after

The largest component of the tax is the engineer context switch, not the retry minutes. Automated recovery that reruns before a red check ever reaches the pull request protects focus, which is why the biggest saving in the model is the move from manual to automated retry.

Run recovery on managed runners to shrink the residual compute tax

Once self-healing removes the human cost, price the recovery minutes cheaply too. A managed runner bills the retry at roughly 70% below the hosted blend, so the same mechanism that protects focus also makes the surviving compute cost small.

Outlook

Expect the flaky tax to become a tracked metric rather than a tolerated cost over the next two years. As more teams instrument first-attempt pass rate and tie it to the kind of dollar and hour figures in this index, flakiness moves out of the category of universal grumbling and into the category of funded reliability work. The teams that make the number visible will act on it; the teams that watch only eventual pass rate will keep paying a tax they cannot see and therefore cannot justify removing.

The architectural answer is converging on self-healing at the runner layer rather than reliability heroics at the test layer. Because the root-cause distribution is dominated by mechanical, transient failures, the highest-leverage intervention is infrastructure that recognizes a transient signal and reruns on a fresh environment automatically. That moves the fix off the critical path of individual engineers and into a platform behavior that pays back on every flake, every day, without competing for the test-suite attention that genuine races still need.

For most teams the practical takeaway is that the flaky tax is both larger and more removable than it looks. It is larger because the human half dwarfs the compute half and hides in slipped estimates rather than on the invoice. It is more removable because the dominant causes are mechanical and yield to automated recovery rather than a test-rewrite marathon. Measure it, route the transient majority through self-healing managed runners, and the index that looked like an immovable cost of doing business comes down sharply, while the engineering experience improves at the same time.

Methodology

This index models the monthly flaky tax as wasted re-run minutes priced at the published GitHub-hosted Linux rate (0.008 per minute) plus engineer context-switch hours valued at a typical loaded rate, summed across a team's monthly pipeline runs. Flake rates, per-team-size run volumes, and context-switch hours are Latchkey analysis (modeled) from typical pipeline shapes, consistent with roughly 14% of billed minutes going to flaky re-runs, and are intended to show direction and magnitude rather than a precise population value. The self-healing managed line applies automated transient-failure retry at the Latchkey rate of 0.0025 per minute, a 70% reduction versus the blended hosted average. DORA reliability bands (change-fail 0-15%, recovery less than one hour) are published by the DORA program; CI adoption near 76% is from the developer survey. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• DORA State of DevOps Report
• Stack Overflow Developer Survey
• GitHub Actions - billing & pricing
• GitLab DevSecOps Survey

More Latchkey reports