Skip to content

feat(serve-logging): reproducible orca-serve logging & diagnostics pack - #39

Merged
LesleyMurfin merged 22 commits into
mainfrom
feature/serve-logging-setup
Sep 17, 2026
Merged

LesleyMurfin merged 22 commits into
mainfrom
feature/serve-logging-setup

Conversation

@LesleyMurfin

@LesleyMurfin LesleyMurfin commented Sep 11, 2026 •

Copy link
Copy Markdown
Owner
Files Added Deleted Net
Test 9 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​346 $\color{#cf222e}{\Huge{\mathbf{−}}}$​5 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​341
Prod 27 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​2808 $\color{#cf222e}{\Huge{\mathbf{−}}}$​21 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​2787

Summary

Reproducible Orca Serve logging & diagnostics pack — one script (install-logging-setup.sh) provisions the systemd instance unit, config layer, per-instance state tree, and log rotation + journal retention from templates. Plus the orca-serve-troubleshoot skill (install + 4-bucket triage). It arranges the observability surface only — it does not download, build, or start the serve.

In addition, this PR resolves Fork Issue #23 / PRB-0015: Xvfb stale display lock self-healing on headless Linux serve, fixing browser pane crashes caused by leftover /tmp/.X99-lock files.

+----------------------------------------------------------------------------------+
|                              WHAT THIS PR ADDS                                   |
|                                                                                  |
|  docs/reference/serve-logging/                                                   |
|    templates/install-logging-setup.sh        one-command installer / auditor     |
|    templates/orca-serve@.service.template    systemd instance unit (@PREFIX@ etc)|
|    templates/orca-serve.conf.template        host-wide config (shared)           |
|    templates/orca-serve-instance.env.template per-instance config (overrides)    |
|    templates/orca-client-diagnostics.md      client-side pairing runbook (Bkt 4) |
|    orca-serve-logging-guide.md               architecture + env vars + sinks      |
|    orca-serve-troubleshooting-matrix.md      symptom -> bucket -> <=5 cmd triage |
|  skills/orca-serve-troubleshoot/SKILL.md     install + triage modes              |
|  tests/docker/Dockerfile.serve-logging-test  containerized verification suite    |
|  tests/docker/run-test-in-container.sh       isolated end-to-end container runner|
|  tests/docker/Dockerfile.xvfb-test          stale Xvfb lock test container      |
|  tests/docker/run-xvfb-test.sh               isolated Xvfb stale lock verification|
|  src/main/startup/ensure-virtual-display.ts  Xvfb stale lock detection & cleanup  |
+----------------------------------------------------------------------------------+

Current State

Future State

  • One --dry-run-first, idempotent installer reproduces the whole observability surface on any Linux/systemd host.
  • Every installed artifact is placeholder-free; SERVICE_USER/SERVICE_GROUP templating propagates to the unit.
  • --verbose flag & cross-repo portability: Installer supports detailed preflight diagnostics and customizable source repository/branch URLs via ORCA_RAW_REPO_BASE / SKILL_RAW_URL to allow seamless installation across forks and upstream mirrors.
  • Host Divergence Matrix & Action 0: Troubleshooting guide and agent skill include an environment fingerprinting step (Action 0) and explicit divergence matrix handling traps across Ubuntu bare-metal, WSL2, Docker/containers, and macOS Darwin.
  • Containerized Test Harnesses: Added reproducible containerized test actions:
    • tests/docker/Dockerfile.serve-logging-test and run-test-in-container.sh: verifies sandbox installation, dry-run auditing, idempotency, logrotate syntax, and systemd-analyze verification.
    • tests/docker/Dockerfile.xvfb-test and run-xvfb-test.sh: verifies Xvfb stale display lock cleanup, orphan lock reap on dead PID / ESRCH, and self-healing headless serve startup under Ubuntu Linux container conditions.
  • Xvfb Stale Lock Self-Healing (Fork Issue [Bug]: Headless serve self-spawned Xvfb doesn't reap a stale /tmp/.X99-lock after an unclean exit (reaper is socket-gated) → serve can't get its virtual display and crash-loops #23 / PRB-0015): isStaleDisplayLock decouples stale lock detection from unix socket presence, inspecting /tmp/.X${display}-lock and testing PID liveness via process.kill(pid, 0). Stale locks from dead/corrupt PIDs are cleanly unlinked prior to spawning Xvfb.

Testing

Tested end-to-end across environments:

  • bash -n, --help, --dry-run, and --verbose — pass
  • sandbox install (all dirs → /tmp) — pass, no leftover @..@ in any artifact
  • systemd-analyze verify — exit 0
  • logrotate -d — exit 0
  • idempotency re-run — keep (already present), exit 0
  • skill triage vs live orca-serve@factory — active, :6768 LISTEN
  • containerized testing harness in tests/docker/ (serve logging & Xvfb stale lock suites)
  • Unit & regression test suite:
    • vitest run src/main/startup/ensure-virtual-display.test.ts — 38/38 passing
    • oxlint src/main/startup/ensure-virtual-display.ts — 0 errors
    • typecheck:node (tsc -p config/tsconfig.node.json) — clean (0 errors)

Provenance

  • Workspace: lesleyMurfin/orca — branch feature/serve-logging-setup
  • Hosts tested: mtl-02-dev-001, Lesleys-PC (WSL2), Docker container

Documentation

Upstream Issue & PR Linkages

Follow-ups

  • Create the orca service account on hosts before starting the unit (installer soft-warns; full serve installer provisions it).
  • systemd-analyze verify flags the missing orca binary — expected (installer does not provision it).
  • journald retention drop-in is inert until systemctl restart systemd-journald.

Refs: serve-logging standardization, Fork Issue #23, PRB-0015

Ticket: INFRA

LesleyMurfin and others added 9 commits September 10, 2026 10:03
Port the headless `orca serve` logging/diagnostics pack from revive_labs:
- logging architecture guide + troubleshooting matrix
- reproducible templates (systemd unit, config/env layers, installer, client diagnostics)
Link the guides from README next to the headless-linux-server guide.
…stream surface

Rewrite the revive_labs port so it references only what upstream orca actually
ships: real `orca serve` flags, the `orca_server_ready` ready contract, the
`runtime.sock` local transport, real env vars (ORCA_ENVIRONMENT, ORCA_VERSION,
ORCA_USER_DATA_PATH), and journald/logrotate logging. Drop fork-only tooling
(orca-serve-fg/-diag/-version, OSV_*/ORCA_SERVE_* vars, /data/opt/revive, svc_orca).
@coderabbitai

coderabbitai Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Review Change StackReview Change Stack

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 313910eb-356f-41ce-b9bc-0bb24a149a57

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/serve-logging-setup

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Decouple stale Xvfb lock cleanup from socket existence in ensureVirtualDisplayForHeadlessServe.
Add isStaleDisplayLock to inspect /tmp/.X${display}-lock and probe PID liveness via process.kill(pid, 0).
Fixes Fork Issue #23 (PRB-0015).
Verified with unit tests and Docker Ubuntu 24.04 containerized test harness.
@LesleyMurfin
LesleyMurfin merged this pull request into main Sep 17, 2026
31 of 48 checks passed
@LesleyMurfin
LesleyMurfin deleted the feature/serve-logging-setup branch September 17, 2026 22:47
@LesleyMurfin
LesleyMurfin restored the feature/serve-logging-setup branch September 18, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant