Elasticsearch Stops Indexing: Disk Watermarks Exceeded (flood_stage)

ES protects itself: past the flood-stage disk watermark (95% by default) it blocks writes and marks indices read-only. Clear disk space and unlock the indices — in that order.

What you'll see

Root causes

Disk usage crossed low→high→flood watermarks

Defaults: 85% low (no new shards), 90% high (relocate shards), 95% flood (block writes, mark read-only). df -h on the data nodes shows where you stand.

Indices keep their read_only_allow_delete block after cleanup

Flood stage sets index.blocks.read_only_allow_delete=true; freeing disk does NOT auto-clear it on older versions — you must remove the setting.

Fix it

  1. See per-node disk and the watermark verdict
    curl -s localhost:9200/_cat/allocation?v && df -h /var/lib/elasticsearch
  2. Free real space (old indices, snapshots, logs)
    curl -s -XDELETE 'localhost:9200/logs-2025.05-*'   # or use ILM/curator to prune automatically going forward
  3. Unlock flood-blocked indices
    curl -s -XPUT localhost:9200/_all/_settings -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete": null}'
  4. Prevent recurrence with watermarks matched to your disks
    # elasticsearch.yml: cluster.routing.allocation.disk.watermark.low: 85%, high: 90%, flood_stage: 93%  (plus monitoring/alerting at ~80%)

Field note

Big wins: check _cat/indices?v sorted by store.size for forgotten debug indices; enable ILM rollover so time-series data prunes itself. On 7.4+ flood blocks usually auto-clear, but verify the setting is null after space frees — assuming is how outages repeat.

Common questions

I deleted data — why is indexing still blocked?

The read_only_allow_delete block often persists after space frees (version-dependent). Remove it explicitly with the _settings PUT shown above, then confirm with GET <index>/_settings.

What's a safe disk ceiling for Elasticsearch nodes?

Plan capacity so steady-state stays under the low watermark (85% default). Once you're tuning flood_stage upward instead of freeing space, you're borrowing an outage.

Related fixes

Ship it right the first time

Our most-documented failures, packaged as ready-to-ship starter kits: Docker, Kubernetes, and Terraform.

Browse the template store →

One-time. Yours to modify. Instant download from the NinjaOps template store.