The State of CI/CD for Data Teams 2026

How data and analytics teams test pipelines, run notebook CI, and keep slow validation from blocking every change to the warehouse.

Executive summary

Data teams adopted CI later than application teams, and it shows in the shape of their pipelines. A model change or a transformation edit cannot be validated by a fast unit test alone. It has to run against representative data, materialize tables, and check freshness, uniqueness, and referential integrity. That makes data CI slow by nature: validation runs are measured in tens of minutes, not seconds, and they sit squarely on the critical path of every change to the warehouse. The slowness is not a bug in the tooling, it is a property of validating against real data, which is exactly what makes the validation worth doing.

On top of slowness sits a particular kind of flakiness. Data pipelines fail for reasons application tests rarely encounter: an upstream source arrived late, a warehouse connection timed out under contention, a sampled dataset shifted under a threshold between runs. Many of these failures are transient and pass cleanly on a retry, which means a data team that re-runs them by hand is paying the same flaky-test tax that application teams already learned to automate away years ago. The difference is that data teams are often still paying it manually, one re-triggered pipeline at a time.

There is a second failure mode that is not transient at all: schema and contract drift. An upstream column changes type, a source drops a field, a join silently widens. These are real breakages, and catching them in CI rather than in a production pipeline at 2am is one of the highest-value things a data team can do. Reliability for data teams is therefore two things working together: contracts that catch genuine drift early, and runners that do not manufacture false failures out of transient infrastructure hiccups.

This report quantifies where data-CI time goes, why validation runs are long, what actually causes data-CI runs to fail, and how notebook execution in CI lives or dies on environment caching. It then shows why bursty, memory-hungry, transient-failure-prone data workloads are an unusually good fit for managed self-healing runners that scale up for the validation burst and back to zero between runs.

The throughline is that data CI is slow because the data is real, so the win is never to skip validation. The win is to make validation scoped to what changed, parallel across independent models, cached at the environment layer, and self-healing against transient infrastructure failures. Each of those is mechanical, and together they turn a multi-tens-of-minutes blocker into something a data engineer can run on every pull request without dreading the wait.

Validation runs sit on the critical path and they are long

A data change is not done until it has been validated against representative data, and that validation is dominated by materialization: actually building the tables and running freshness, uniqueness, and referential-integrity checks against them. The earlier stages of a data pipeline are fast. Linting and SQL compilation take a couple of minutes, unit and macro tests a few more, but the moment the pipeline materializes real tables the wall-clock time jumps.

The stage chart makes the distribution clear. Lint and compile are cheap, unit and macro tests modest, data tests on fixtures more substantial, and full materialization is the single largest stage by a wide margin. This is why data-CI run times are measured in tens of minutes rather than seconds, and it is why a data engineer feels the pipeline as a tax in a way an application engineer with a sub-minute test suite does not.

The leverage is not in skipping the materialization, which is the part that actually catches problems. It is in scoping the materialization to the models a change genuinely affects rather than rebuilding the whole DAG, and in running independent models in parallel rather than serially. A one-line change to a single mart model should validate that model and its downstream dependents, not the entire warehouse, and the independent branches of the DAG should run at the same time on separate runners.

• Lint and compile are cheap; full materialization is the dominant stage by wall-clock time.
• Scope materialization to the models a change affects rather than rebuilding the whole DAG.
• Run independent models in parallel across runners instead of serially down the DAG.

A third of data-CI failures are transient, not real

Data pipelines fail for infrastructure reasons application tests rarely hit. A late upstream source, a warehouse connection timeout under contention, a sampled set that drifted under a threshold between runs: these produce red checks that have nothing to do with the change under test. Our model attributes roughly a third of failed data-CI runs to transient causes that pass cleanly on a retry, the largest single category of failure.

The failure-cause chart separates the transient slice from the genuine ones. Transient source and connection timeouts lead, followed by genuine data-quality assertions, schema and contract drift, and resource limits. The important read is that the single biggest cause of a red data-CI run is not a real defect, it is the warehouse or an upstream system having a bad moment, and a re-run would have gone green.

Re-running these by hand is the same flaky-test tax application teams already automated away, and it is the single clearest case for self-healing runners on a data team. When a step fails on a known-transient signal, a connection timeout or an OOM kill on an oversized sample, the platform retries it on a fresh environment automatically, before a data engineer ever sees a red check or loses focus to investigate a failure that was never real. The genuine failures, the data-quality assertions and the contract drift, still surface, because those are the ones a human needs to see.

• The largest single cause of failed data-CI runs is transient, not a real defect.
• Manual re-runs of transient failures are the flaky-test tax data teams often still pay by hand.
• Auto-retry on transient signals while letting genuine data-quality and contract failures through to a human.

Notebook CI lives or dies on environment caching

Executing notebooks in CI to catch broken analyses is valuable: it stops a colleague from inheriting a notebook that no longer runs top to bottom. But it is punishing to do naively, because the scientific Python and R stacks are heavy and slow to install. A notebook that runs in seconds can sit behind several minutes of environment setup, and if that setup happens from scratch on every run, the pipeline becomes something engineers route around rather than rely on.

The environment chart shows the spread. A cached-dependency warm runner executes the suite quickly, a cached-dependency cold runner takes longer, uncached dependencies take longer still, and a full environment rebuild is several times the warm case. The gap between the best and worst strategy is the whole story: the notebook code is the same in every column, only the environment handling differs.

The discipline is the same caching discipline that pays off everywhere in CI, applied to an unusually heavy dependency stack. Keys scoped to the lockfile and the interpreter version, a warm runner that already holds the common packages, and a cache that is measured rather than assumed. Notebook CI that does not cache its environment is notebook CI that the team will quietly abandon, taking the broken-analysis safety net with it.

Pipeline reliability is a data-contract problem as much as a runner problem

Beyond transient failures, a meaningful share of data-CI breakage is schema and contract drift: an upstream column changed type, a source dropped a field, a join silently widened its grain. These are not infrastructure hiccups and they should not be retried away. They are real defects propagating from upstream, and the only question is whether they are caught in a pull-request check or in a production pipeline page in the middle of the night.

Catching them in CI with explicit data contracts moves the failure left. A contract that asserts the expected columns, types, and grain of a source turns a silent downstream corruption into a loud, specific failure on the pull request that introduced the dependency on the changed shape. This is the data-team analogue of a type system: it makes a class of integration bug impossible to merge rather than merely likely to be noticed eventually.

Reliability for data teams is therefore the combination of two things that are easy to confuse. Contracts catch the real drift early, and self-healing runners stop the transient infrastructure failures from manufacturing false alarms. A team that has only contracts still drowns in transient red checks, and a team that only auto-heals still ships contract drift to production. The reliable teams do both, so that every red check that reaches a human is one worth their attention.

Memory limits are a quiet, recurring source of failure

Data jobs are memory-hungry in a way most application CI is not. A transformation that runs fine on a sampled fixture can be killed for running out of memory on a larger set, and the failure is intermittent because it depends on the size of the data that happened to flow through that run. An out-of-memory kill looks like a crash, sends a red check, and costs a full re-run, which is why it shows up as its own slice of the failure breakdown rather than folding into the transient bucket.

There are two complementary fixes, and they pull in different directions, so the right answer is to use both deliberately. The first is to right-size the runner for the job: a materialization over a large fixture needs more memory than a lint step, and defaulting everything to a small runner guarantees periodic OOM kills on the heavy stages. The second is to scope and sample fixtures so that CI validates correctness without trying to process production-scale volumes on every pull request.

Elastic managed runners make the right-sizing half cheap, because the larger runner is provisioned only for the heavy stage that needs it and released afterward rather than paid for around the clock. Combined with auto-recovery for the genuinely transient OOM kills, this turns a recurring, irritating failure into a non-event that the platform absorbs without a human in the loop.

Managed self-healing runners fit data workloads especially well

Data jobs are bursty, memory-hungry, and prone to transient infrastructure failures, which is the exact profile managed runners handle best. They scale up for the validation burst when a batch of pull requests lands, self-heal the timeout-and-retry failures that dominate the failure breakdown, and scale back to zero between runs instead of charging for idle capacity that sits unused most of the day.

The economics reinforce the fit. At a modeled $0.0025 per minute against GitHub-hosted Linux at $0.008 per minute, managed runners run roughly 70 percent cheaper, and that gap applies precisely to the long materialization stages where data CI spends most of its minutes. A workload that spikes and then goes quiet is the worst possible fit for a fixed fleet sized for peak, and the best possible fit for elastic capacity that costs nothing when idle.

The deciding factor for most data teams is that they do not have a platform group to size, patch, and babysit a runner fleet. Managed runners remove that operational burden entirely while delivering the right memory for the heavy stages, automated recovery for the transient failures, and a per-minute rate well below hosted. For a team whose pipelines are slow because the data is real, removing the mechanical waste around the validation is the highest-return move available.

• Data workloads are bursty, memory-hungry, and transient-failure-prone, the ideal managed-runner profile.
• Managed runs at a modeled $0.0025/min versus hosted Linux at $0.008/min, roughly 70% cheaper.
• Elastic capacity costs nothing between validation bursts, unlike a fixed fleet sized for peak.

What the fastest data teams do differently

The data teams that keep validation fast despite real data do not skip the validation. They share a short set of habits that scope, parallelize, cache, and self-heal the pipeline so that the unavoidable cost of materializing real tables is paid only where it adds value. None of these habits require rewriting the transformations, and most are configuration plus the right runner layer underneath.

They also treat the pipeline as a measured surface. They watch validation wall-clock time, the transient-failure rate, cache hit rates on the notebook and dependency layers, and the share of red checks that turn out to be real, and they treat regressions as bugs to fix. Because they can see the curve, they notice a creeping materialization scope or a cache that stopped hitting before it turns the pipeline into something the team dreads running.

• Scope materialization to affected models and run independent models in parallel.
• Define data contracts so genuine schema and grain drift fails on the pull request, not in production.
• Cache the heavy notebook and dependency environments, with monitored hit rates.
• Auto-heal transient timeouts and OOM kills so false red checks never reach a human.
• Run on elastic managed runners that scale to the burst and right-size memory per stage.

Recommendations

Scope and parallelize materialization

Make validation do work proportional to the change rather than to the warehouse. Materialize the models a change affects plus their downstream dependents, not the whole DAG, and run independent branches in parallel across runners. This is the single largest lever on the long materialization stage that dominates data-CI wall-clock time.

Define data contracts to catch drift on the pull request

Assert the expected columns, types, and grain of upstream sources so that schema and contract drift fails loudly on the pull request that introduces the dependency rather than silently corrupting a production table. Contracts are the data-team type system, and they move a costly class of failure from a 2am page to a CI check.

Cache the notebook and dependency environment, and measure hit rate

The scientific Python and R stacks are heavy, so a cold or uncached environment dwarfs the notebook execution itself. Key the cache to the lockfile and interpreter version, keep a warm runner that holds the common packages, and watch the hit rate the way you watch test coverage so a silently broken cache does not creep the run time back up.

Auto-heal transient failures instead of re-running by hand

The largest single cause of failed data-CI runs is transient: source timeouts, connection drops, OOM kills on oversized samples. Retry these automatically on a fresh environment so they never reach a data engineer, while letting genuine data-quality assertions and contract failures through to a human who needs to see them.

Run on elastic managed runners sized per stage

Data workloads spike for the validation burst and go quiet between runs, the worst fit for a fixed fleet and the best fit for elastic capacity that costs nothing when idle. Provision more memory only for the heavy materialization stages that need it, and let the runner layer scale to the burst at roughly 70% below hosted rates.

Outlook

Expect data CI to keep converging on the patterns application teams settled years ago, adapted to the reality that data validation is intrinsically slow. The teams that scope materialization, parallelize the DAG, cache heavy environments, and auto-heal transient failures will run validation on every pull request without flinching, while the teams that still re-trigger failed pipelines by hand and rebuild the whole DAG on every change will keep treating CI as a tax to avoid rather than a safety net to rely on.

Data contracts will become a default rather than an advanced practice. As warehouses grow more interconnected and more teams depend on shared upstream models, silent schema and grain drift becomes too expensive to catch in production, and the pull-request check that fails loudly on a contract violation will be standard hygiene rather than a sophistication. Reliability will increasingly be understood as contracts plus self-healing runners working together, not as either one alone.

For most data teams the practical takeaway is that the slowness of validating against real data is not the enemy and should not be optimized away by skipping checks. The enemy is the mechanical waste around it: unscoped materialization, serial DAG runs, uncached environments, and manually re-triggered transient failures. A runner layer that scales to the burst, right-sizes memory, caches the heavy stages, and heals the transient failures removes that waste, leaving only the validation that is genuinely worth the wait.

Methodology

This report synthesizes publicly available industry data (developer surveys and the DORA State of DevOps research) with Latchkey's own analysis of data-team pipeline economics. Validation run times, failure-cause splits, and notebook-CI execution times are modeled from typical analytics-engineering and notebook pipeline shapes, plus published runner pricing, and are labeled as such rather than drawn from a primary survey of named teams. Runner rates and the managed savings figure reconcile with the shared Latchkey pricing spine. Figures labeled "modeled" are illustrative estimates derived from public pricing and typical pipeline shapes, not a primary survey; figures attributed to a named source reflect that source. Pricing reflects published rates at time of writing and should be verified against current provider pricing.

Sources

• Stack Overflow Developer Survey
• DORA State of DevOps Report
• JetBrains Developer Ecosystem Survey
• GitHub Actions - billing & pricing

More Latchkey reports