Repository navigation
claude-ops observability: make OTEL retention visible and size-bounded #5236
Description
Problem
The claude-ops OTEL store prune depends on an operator-registered scheduled task, and nothing in the plugin shows whether it is running or whether the store is growing. See provisioning issue melodic-software/provisioning#661 for registering the task.
Evidence on melo-lap-001 (2026-09-28): no prune task existed. Hot cc-logs.json reached 2.8 GB and cc-metrics.json 361 MB.
ADR-0009's cold-tier estimate ("tens of MB/month") is stale against measurements: cold held 72 MB for about 5 weeks (1.38M log rows, 2026-08-16 to 2026-09-22), and the hot tier held 790 MB of logs for 7 days because log lines carry inline API bodies.
The collector's fileexporter rotation cannot be combined with append: true (collector-contrib fileexporter README), so an external prune stays necessary.
Proposed change
- The
statusaction reports hot store size, cold store size, and last-prune time and age. - Add a size-cap fallback to
prune-otel-store.sh: prune when a hot file exceeds N MB regardless of age (N configurable, with a documented default). - Refresh ADR-0009's cold-size estimate and the hot-tier sizing with the measurements above.
- Note the fileexporter rotation/append constraint in the docs so nobody re-opens it.
Acceptance criteria
statusprints hot size, cold size, and last-prune time/age.prune-otel-store.test.shcovers the size-cap path (a hot file over the cap is pruned even when younger than the age limit).- ADR-0009 and the retention docs match the measured numbers.
Activity
kyle-sexton commentedon Sep 29, 2026
Additional bug found while running the prune manually on melo-lap-001 (2026-09-28):
prune-otel-store.shfails under Git Bash whenCC_OTEL_STOREholds the machine value as a backslash Windows path (C:\ProgramData\local-otel\cc-store). awk reports "cannot redirect toC:ProgramDatalocal-otel..." because the backslashes are consumed. Nothing was mutated and the service kept running. Re-running withCC_OTEL_STORE=C:/ProgramData/local-otel/cc-storesucceeded. The script should normalize the path (for examplecygpath -mor replacing backslashes) before using it. Any scheduled task that calls it with the provisioned machine variable will hit this, so it blocks provisioning#661.feat(claude-ops): report cold size and last-prune age in the OTEL sto…
chore(claude-ops): bump to 0.65.0 with the OTEL retention changelog e…
feat(claude-ops): OTEL store size cap and retention visibility (#5425)
kyle-sexton commentedon Oct 1, 2026
Owner decision (2026-10-01): approved the recommendation ("go with recommended", the owner's reply to the batch decision page that listed this question and its recommendation).
Question: the only open criterion on #5236 is the ADR-0009 sizing refresh, which lives in melodic-software/medley. Chosen option A: an agent opens a draft PR in medley that appends a dated amendment (2026-10-01) to docs/adr/0009-local-claude-code-observability-backend.md. It records cold at 72 MB over about 5 weeks (1.38M log rows), hot logs at 790 MB per 7 days, the CC_OTEL_HOT_MAX_MB cap (default 1024), and that fileexporter rotation cannot be combined with append.
Next: an agent opens the draft medley PR and links it here; #5236 closes when that PR merges. Relabeled agent-ready.
kyle-sexton commentedon Oct 1, 2026
Draft medley PR for the ADR-0009 amendment: https://github.com/melodic-software/medley/pull/2079. #5236 closes when it merges; the merge is the owner's call. Issue stays open.
kyle-sexton commentedon Oct 1, 2026
All three acceptance criteria hold on main; closing.
Merged:
CC_OTEL_HOT_MAX_MBsize cap inprune-otel-store.sh(default 1024 MiB),probe-observability-state.sh --otel-storeprints hot size, cold size andlast-prune:<time> (<age>), retention docs carry the measured sizes and the fileexporter rotation/append constraint.CC_OTEL_STOREpaths.## Amendment (2026-10-01)section with cold 72 MB over about 5 weeks (1.38M log rows), hot logs 790 MB per 7 days, theCC_OTEL_HOT_MAX_MBcap, and the fileexporter v0.153.0 rotation/append constraint.Criteria:
statusprints hot size, cold size and last-prune time/age: met (probe-observability-state.sh lines 230-247).prune-otel-store.test.shcovers the size-cap path: met (case 26, a hot file over the cap is pruned though every line is young).Not claimed: the ADR amendment does not re-measure the 2026-06-09 "~2.6 GB to ~1 GB steady-state hot store" estimate; it states only the measured 7-day volume.