CockroachDB "restart transaction: TransactionRetryError" in CI

By Daniel Zoghalchali·Latchkey

CockroachDB uses SERIALIZABLE isolation. When concurrent transactions conflict, it aborts one with a retryable error and expects the client to retry. Tests that run transactions in parallel can hit this nondeterministically in CI.

What this error means

A transaction fails with "restart transaction: TransactionRetryError: ... RETRY_SERIALIZABLE" (SQLSTATE 40001), often only on some runs, which looks like flakiness.

ERROR: restart transaction: TransactionRetryWithProtoRefreshError:
TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh)
SQLSTATE: 40001

Common causes

Concurrent transactions contend for the same rows

Two transactions read and write overlapping data; under SERIALIZABLE one must restart to preserve isolation.

The client does not retry 40001

Code that treats the retryable error as fatal fails the job instead of re-executing the transaction.

How to fix it

Retry transactions on SQLSTATE 40001

1. Catch the 40001 retry error around the transaction.
2. Re-run the whole transaction body, not just the failed statement.
3. Cap retries with backoff to avoid an infinite loop.

for attempt in range(5):
    try:
        with conn.transaction():
            run_statements()
        break
    except psycopg.errors.SerializationFailure:
        time.sleep(0.1 * (attempt + 1))

Reduce contention in test fixtures

Give parallel tests separate rows or databases so transactions do not collide as often.

How to prevent it

• Wrap transactions in a retry helper that handles 40001.
• Isolate parallel test data to cut contention.
• Treat retry errors as expected, not as test failures.

Frequently asked questions

Related guides

References

• GitHub Actions documentation