Data quality checks: threshold and tolerance failures in CI

By Daniel Zoghalchali·Latchkey

A data quality check fails because it demands zero defects while the data has a known, acceptable small rate. The fix is a realistic tolerance in the check, not silencing the failure.

What this error means

Checks fail on a small percentage of expected exceptions (a few nulls, a rare category) and block CI, even though the rate is within business tolerance.

missing_count(secondary_email) = 87 [FAILED]
  check: missing_count(secondary_email) = 0
  (secondary_email is optional; ~1% missing is expected)

Common causes

A zero-defect check on tolerant data

The check demands exactly zero when the column legitimately has a small missing or exceptional rate, so it fails on normal data.

No tolerance expressed for known variance

The business accepts a small rate but the check encodes an absolute rather than a percentage or threshold.

How to fix it

Express a percentage or threshold

1. Decide the acceptable rate with the data owner.
2. Change the check from an absolute to a percentage or mostly threshold.
3. Keep it failing hard past the tolerance so real regressions still trip.

checks for customers:
  - missing_percent(secondary_email) < 5:
      name: secondary_email mostly present

Use mostly / tolerance in the validation tool

Great Expectations mostly, dbt test config: {error_if, warn_if}, and Soda thresholds all express tolerance without deleting the check.

expect_column_values_to_not_be_null(column="secondary_email", mostly=0.95)

How to prevent it

• Set tolerances with the data owner, not by trial and error.
• Prefer percentage or threshold checks over absolute zero for tolerant columns.
• Keep hard failures for truly required columns.

Frequently asked questions

Related guides

References

• GitHub Actions documentation