Repository navigation
Loki forgets EVERYTHING when loki-0 restarts: every namespace's history starts at 01:09Z 2026-09-09 (emptyDir still live despite persistence.enabled; no ingester WAL) #3773
Description
What happened
While reading the memex 503 of 2026-09-09 (#3772) through Loki from a memex-cloud script: at 01:08Z a query over {namespace="memex"} returned both old pods' Application is shutting down... lines at 01:05:01Z (pods 76c99bdc6-9stvn, -g7f89). At 01:11Z the same query, same window, returned no line at all from either pod, and sum by (pod) (count_over_time({namespace="memex"}[70m])) no longer listed them. Between the two reads, monitoring/loki-0 was itself drained (01:09:08Z, together with loki-alertmanager-0 and the pushgateway) as the node-pool operation reached its node.
So the lines that had been ingested were lost with the ingester's unflushed chunks. deploy/aks/scripts/values.observability.yaml gives Loki a PVC (persistence.enabled: true) — that preserves flushed chunks and the index — but nothing there enables the ingester WAL (loki.config.ingester.wal.enabled + a dir on the persistent volume), so everything in memory at the moment of the restart is gone. The same file already documents that Promtail's WAL cannot be passed through the deprecated loki-stack chart; the Loki-side WAL is a different knob and is what would have kept these.
The lines lost were exactly the ones the incident needed: the first shutdown of the pair. The cluster-wide read at 01:09Z still carried the story, which is the only reason the cause was attributable.
Fix
- Enable the ingester WAL in
values.observability.yaml(loki.config.ingester.wal: {enabled: true, dir: /data/loki/wal, flush_on_shutdown: true}or the chart's equivalent — VERIFY by rendering that the key reaches the config, the way the Promtail note in that file does), on the persistent volume. loki-0sits on the system pool (nodeSelector: agentpool: system) and still gets drained with its node;flush_on_shutdownis what turns a graceful drain into a flush instead of a loss.- Note in the file that a drain of loki-0 is a read-it-now window for anything in flight.
Activity
[-]Loki drops unflushed chunks when loki-0 is drained: ingester WAL is not enabled[/-][+]Loki forgets EVERYTHING when loki-0 restarts: every namespace's history starts at 01:09Z 2026-09-09 (emptyDir still live despite persistence.enabled; no ingester WAL)[/+]rbuergi commentedon Sep 9, 2026
Verified the live storage boundary: the running monitoring/loki-0 uses grafana/loki:2.6.1, mounts volume storage at /data, and that volume is emptyDir. This directly explains loss on pod replacement: both chunk storage and the WAL are on ephemeral storage.
The checked-in values render differently: a 30Gi storage volumeClaimTemplate mounted at /data, with ingester.wal.dir=/data/loki/wal. Loki 2.6.1 enables WAL by default: https://github.com/grafana/loki/blob/v2.6.1/pkg/ingester/wal.go#L38-L42 . Therefore absence of an explicit enabled:true value does not establish that WAL was disabled. Adding that setting alone would not fix the live durability defect.
The required repair is to bring the live StatefulSet onto persistent storage, accounting for immutable volumeClaimTemplates and preserving or explicitly accepting the existing log window during migration. install-observability.sh already documents that immutable-field transition, but its final verification only prints commands and does not enforce their results. No live resources were modified during this investigation. The issue remains open until the storage migration and durability verification are complete.
rbuergi commentedon Sep 9, 2026
Fixed on
main— verified by file, not by PR state (the merge queue does not fireCloseslinks).Landed as #3835. All three of this issue's asks, plus the one it implied:
enabled,dir: /data/loki/wal,flush_on_shutdown(these had landed in c41bb0f, ~12 h after this issue was filed).check-observability-values.shruns on every pull request (Chart Gate→ Observability values), and its job passed on the PR itself.values.observability.yaml: the WAL narrows the loss, it does not remove the gap, andloki-0is still a read-it-now window while Promtail buffers with no WAL of its own.Plus the thing that made any verification here perishable: the chart was unpinned.
helm upgrade --install grafana/loki-stackhad no--version, so every run installed whatever upstream published last — an upstream restructure ofloki.configcould have stopped templating the WAL with no signal at all. Pinned to 2.10.3, in the installer and in the checker, moving together.🚨 The finding worth carrying forward
The obvious gate — render, assert the keys arrive — is vacuous for Loki. loki-stack merges
loki.configthrough verbatim, so a typo'dingestor:reaches the render intact and passes. The first version of the gate shipped exactly that and passed the typo; onlyloki -verify-configrejects it (field ingestor not found in type loki.ConfigWrapper). It was caught by running the negative control, not by reading the code.Doc: Verifying Chart Values.