Cassandra "schema agreement ... timed out" during migrations in CI
After a DDL statement, the driver waits for every node to agree on the new schema version before returning. If agreement is not reached within the timeout, it raises a schema-agreement error. In CI this often follows running migrations against a node that is still settling.
What this error means
A CREATE TABLE or ALTER step warns or fails with "Schema agreement not reached" or "Cluster schema agreement ... timed out", and subsequent statements may see the old schema.
cassandra.OperationTimedOut: errors=Cluster schema agreement was not reached
within the timeout, last_host=127.0.0.1Common causes
The node is still stabilizing after boot
A node that just started can be slow to converge on schema, so the agreement check times out even on a single-node cluster.
Rapid back-to-back DDL without waiting
Firing many schema changes in quick succession can outrun the agreement window, especially under load on a small CI runner.
How to fix it
Raise the schema agreement timeout and retry
- Increase
max_schema_agreement_waiton the Cluster. - Apply DDL one statement at a time.
- Retry the migration if agreement times out the first time.
from cassandra.cluster import Cluster
cluster = Cluster(['127.0.0.1'], max_schema_agreement_wait=30)
session = cluster.connect()Wait for the node to settle before DDL
Require the node to be Up/Normal before applying schema so agreement converges quickly.
until nodetool status | grep -qE '^UN'; do sleep 5; done
cqlsh 127.0.0.1 9042 -e "SOURCE 'migrations.cql'"How to prevent it
- Apply DDL serially and wait for agreement between steps.
- Give the node time to reach Up/Normal before migrating.
- Raise the agreement wait on slow shared runners.