Elasticsearch "cluster_block_exception ... index read-only / allow delete" in CI
Elasticsearch crossed the flood-stage disk watermark and put every index into a read-only block to protect the node. Writes then fail with cluster_block_exception and FORBIDDEN/12. On a CI runner this usually means the runner disk filled up, not that Elasticsearch itself grew large.
What this error means
Indexing requests fail with "cluster_block_exception ... blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]" while reads still work. The runner disk is near full.
{"error":{"type":"cluster_block_exception","reason":"index [logs] blocked by:
[FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}Common causes
The flood-stage disk watermark was crossed
When free disk drops below the flood-stage watermark (95% used by default), Elasticsearch blocks writes on all indices to avoid running out of space.
The CI runner disk filled up
Docker layers, caches, and build artifacts on the runner consume the same disk the node sees, tripping the watermark even with a small index.
How to fix it
Free disk and clear the read-only block
- Free space on the runner (prune Docker, remove large build artifacts).
- Once disk is below the watermark, clear the block by resetting the read-only-allow-delete setting.
- Confirm writes succeed again.
curl -X PUT "http://localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d '
{"index.blocks.read_only_allow_delete": null}'Relax watermarks for ephemeral CI clusters
For a throwaway single-node test cluster you can lower or disable the watermarks so a nearly full runner disk does not block indexing.
env:
cluster.routing.allocation.disk.threshold_enabled: "false"How to prevent it
- Free runner disk before starting the Elasticsearch service.
- Disable disk watermarks for ephemeral single-node CI clusters.
- Keep test indices small and delete them between test runs.