Skip to content

tj serve daemon lifecycle: launchd unit written but never loaded, stale OTLP endpoints lose telemetry silently #614

Description

@anshss

Severity: Medium

Where: daemon lifecycle (tj onboard's launchd/systemd unit installation, core/server_state.py, the OTLP endpoint baked into shells by _zshrc_otel_block).

Observed: On a machine audit, ~/Library/LaunchAgents/com.tokenjam.serve.plist EXISTS with RunAtLoad/KeepAlive true, but launchctl list | grep -i tokenjam returns NOTHING — it was written and never loaded. The daemon actually holding the DB lock was a manually-started foreground process; if it dies nothing restarts it. Separately, a second unrelated tj serve --port 9341 was running from a different venv, and ~/.local/share/tj/server.state pointed at a THIRD, dead pid/port. Any terminal whose baked-in OTEL_EXPORTER_OTLP_ENDPOINT targets a dead port exports into the void with no error surfaced anywhere.

Why it still matters: the scheduled transcript catch-up (shipped in #598) closes the data gap for anything Claude Code wrote to disk. But it only runs when the daemon runs — so an unloaded launchd unit means the self-healing itself never fires. It also does nothing for non-Claude-Code OTLP sources, which have no on-disk transcript to reconcile from.

Fix direction: (1) make tj onboard VERIFY the unit is loaded (launchctl list / systemctl is-enabled) rather than merely written, and load it if not; report the verified state in tj doctor. (2) Reconcile to one daemon: detect multiple live tj serve instances and a server.state pointing at a dead pid/port, and say so. (3) Probe the configured OTLP endpoint for reachability and warn when a shell's baked-in endpoint cannot be reached.

Done when: onboard fails loudly (or self-corrects) when the unit is not loaded; tj doctor reports daemon-loaded state, duplicate instances, and endpoint reachability; a test covers a written-but-unloaded unit being detected.

Ref: Found while working on the ingest scheduling fix in #598.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions