fix(compose): Vector silently drops all logs since #1478, and prod compose fails schema validation

[obasilakis]

Summary

Two independent defects in docker-compose.prod.yml (one shared with docker-compose.yml). Both are production-only in effect, both fail silently, and both were found while diagnosing a Docker daemon wedge caused by a full disk.

---

Bug 1 — Vector has been silently discarding ALL log output since #1478 (v0.8.0)

Affects: docker-compose.yml and docker-compose.prod.yml

logs-init chowns /data/logs to 1000:1000 and chmods it 775, so the UID-1000 backend's retention sweep can delete archived originals (#1478):

logs-init:
  command: ["sh", "-c", "chown 1000:1000 /data/logs && chmod 775 /data/logs"]

But the vector service runs as root (no user: directive) with cap_drop: ALL. Dropping all capabilities strips CAP_DAC_OVERRIDE — the capability that normally lets root bypass file permission checks. Root is neither the owner (1000) nor in the group (1000), so it falls through to other on a 775 directory: r-x. No write bit.

Every Vector sink write then fails:

ERROR sink{component_id=local_logs component_type=file}: vector::internal_events::file:
  Unable to open the file. path=b"/data/logs/local-2026-07-27.json"
  error=Permission denied (os error 13) error_code=failed_opening_file
ERROR component_events_dropped: Events dropped intentional=false count=1

Why this went unnoticed for ~3 weeks: a file-sink error does not fail Vector's healthcheck, which only probes the :8686 API. The container reports healthy while writing nothing. On the affected instance, /data/logs had no new files between 2026-07-09 and 2026-07-27 — every platform and agent log in that window is gone.

Note the pre-#1478 files are root:root, which is why this only began when the dir ownership changed.

Fix

group_add: ["1000"] on the vector service. This uses the directory's existing group-write bit rather than restoring a capability, so the cap_drop: ALL posture is preserved.

Vector must stay root for the UID: raw container logs under /var/lib/docker/containers/*/*-json.log are 0640 root:root and are unreadable as UID 1000, so user: "1000:1000" would trade a write failure for a read failure.

Verification

before: 0 output files since 2026-07-09; N "Permission denied" errors per cycle
after : -rw-r--r-- 1 root root 2636479 /data/logs/local-2026-07-27.json
        0 permission errors

---

Bug 2 — docker-compose.prod.yml fails schema validation, and hides a conflicting retention default

Affects: docker-compose.prod.yml

backend.environment is a YAML list, and two separate "#1039 packaging gap" fixes each appended the same trio of variables:

[ 66] LOG_RETENTION_DAYS=${LOG_RETENTION_DAYS:-5}       <- block A
[ 67] LOG_ARCHIVE_ENABLED=${LOG_ARCHIVE_ENABLED:-true}
[ 68] LOG_CLEANUP_HOUR=${LOG_CLEANUP_HOUR:-3}
...
[112] LOG_ARCHIVE_ENABLED=${LOG_ARCHIVE_ENABLED:-true}  <- block B
[113] LOG_RETENTION_DAYS=${LOG_RETENTION_DAYS:-90}
[114] LOG_CLEANUP_HOUR=${LOG_CLEANUP_HOUR:-3}

This is two distinct problems, and only one is visible.

2a — hard failure. LOG_ARCHIVE_ENABLED and LOG_CLEANUP_HOUR are byte-identical across both blocks. Compose requires array items to be unique, so the file does not parse at all:

$ docker compose -f docker-compose.prod.yml config
validating docker-compose.prod.yml: services.backend.environment array items[67,112] must be unique

(Docker Compose v2.21.0. Older versions accepted duplicates silently.)

2b — silent semantic conflict. LOG_RETENTION_DAYS is also duplicated, but with different values (5 vs 90). Because the strings differ, the uniqueness check does not flag it. Deduping only the identical pair would fix 2a and leave this in place. Docker resolves a repeated env key by last-wins, so prod would silently run LOG_RETENTION_DAYS=90 while docker-compose.yml and config.py (COMMUNITY_RETENTION_FLOOR_DAYS = 5) both say 5 — with an operator reading block A believing it is 5.

The conflict direction is "keeps 18x more" rather than "deletes early", so unlike #1638 this is a disk-growth bug, not data loss. That still matters: log_archive_service only deletes originals older than the window, so a 90-day window on a busy instance is a large unbounded-growth contributor.

Fix

Remove block B. Block A is correct: it matches docker-compose.yml and the COMMUNITY_RETENTION_FLOOR_DAYS = 5 code default. Block B's LOG_ARCHIVE_PATH-omission rationale is worth keeping and is folded into block A's comment.

---

Scope

Both fixes are compose-only — no schema change, no migration, no code change, no new config surface.

Existing deployments pick up Bug 1's fix on the next docker compose up -d (Vector is recreated because its service config changed). No data recovery is possible for logs already dropped.