Skip to content

fix(labs,ovk-openclaw): backport admins durability patches - #117

Merged
mashkovd merged 1 commit into
mainfrom
fix/labs-ovk-backport-admins-durability-clean
May 5, 2026
Merged

mashkovd merged 1 commit into
mainfrom
fix/labs-ovk-backport-admins-durability-clean

Conversation

@mashkovd

@mashkovd mashkovd commented May 4, 2026

Copy link
Copy Markdown
Member

Summary

Pre-PR before bumping labs/ovk to 2026.5.2-mctl.1. Copilot review on the closed direct labs bump (PR #115) surfaced that admins-specific durability workarounds were never applied to labs/ovk.

Patch Source PR Risk if missing
A Top-level command/args lock-cleanup wrapper mctl-gitops #96 series container restart with stale .openclaw-runtime-mirror.lock/ in emptyDir → 5 min wait for phantom owner → startup probe kill → infinite crash-loop (admins hit 239 restarts in 22h before #96)
B1 restore-state plugin-runtime-deps S3 purge admins wasted ~1.5GiB S3 + ~5min cold-boot restore time per pod
B2 restore-state lock cleanup find ... *.lock -exec rm mctl-gitops #96 S3-poisoned lock owner.json rehydrated on every boot → same 5-min wedge as A
C Widened startup + readiness probe budgets mctl-gitops #101 startup 5+90×5 = 455s (was 185s); readiness 5×10s (was 3×3s) — labs/ovk CPU ceiling 1500m < admins 2000m, more susceptible
D Widened liveness probe budget mctl-gitops #101 15s × 10 thresh = 5 min tolerance (was ~150s)
E Bonjour plugin disabled in JSON config mctl-gitops #102/#103 Bonjour @ciao announcer can throw UnhandledPromiseRejection: CIAO ANNOUNCEMENT CANCELLED when event loop stalls > 170s, crashing Node

Scope

  • labs/openclaw/values.yaml: +44/-6 (1211 → 1249 lines)
  • ovk/openclaw/values.yaml: +44/-6 (1157 → 1195 lines)
  • Image tag NOT touched — labs/ovk stay on 2026.4.26-beta.1 here. Image bump PRs follow after this merges.

NOT touched (tenant-specific)

CPU/memory limits, env vars (OPENCLAW_GITHUB_ALLOWED_REPOS, OAuth callbacks), envFrom secrets (CoinGlass for labs), MCP server registrations, backstage block (labs has, ovk doesn't), persistence/extraVolumes.

Validation post-merge

ArgoCD syncs ~2 min, Recreate strategy → both labs and ovk pods recreate. Quick checks:

  1. kubectl -n labs get pods -l app.kubernetes.io/instance=labs-openclaw → 3/3 Ready, 0 restarts after settle
  2. Same for ovk
  3. Both must still boot fine on 2026.4.26-beta.1 — wrappers are version-agnostic, lock-cleanup is no-op on clean emptyDir, expanded probe budgets just give more headroom

If either tenant has trouble booting on beta.1 with these wrappers — revert and investigate before any image bump.

Next steps

After this merges and labs+ovk soak briefly with wrappers in place:

  1. Re-open labs-openclaw bump to 2026.5.2-mctl.1 (one-line, equivalent of closed chore(labs-openclaw): bump image to 2026.5.2-mctl.1 #115)
  2. 30-min watch-loop on labs (per playbook)
  3. Same for ovk

Test plan

  • CI green
  • At least one of codex / copilot review clean
  • Post-merge: both tenants 3/3 Ready, 0 restarts, ArgoCD Synced+Healthy

Pre-PR before bumping labs/ovk to 2026.5.2-mctl.1. Copilot review on the
direct labs bump PR #115 surfaced that admins-specific workarounds were
never applied to labs/ovk:

- PR #96 lock-scrub entrypoint wrapper (top-level command/args)
- PR #96 restore-state lock cleanup (find ... *.lock -exec rm)
- PR #101 widened probe budgets (startup 455s, readiness 5x, liveness 5min)
- PR #102/#103 bonjour disable (in JSON config)
- plugin-runtime-deps S3 purge (perf + restore time)

Without these, bumping labs/ovk to 2026.5.2-mctl.1 would re-introduce
the lock-wedge and probe-kill failure modes that admins survived only
because the workarounds were in place.

Tenant-specific differences (CPU limits, OAuth callback URLs, env vars,
MCP server registrations) are intentionally NOT touched — only
durability/safety patches copied over with paths adapted per tenant
('labs' vs 'ovk' in S3 prefixes).

Both files now +38 lines each. Closed PR #115 (labs image bump) — will
re-open after this merges.
@mashkovd

mashkovd commented May 4, 2026

Copy link
Copy Markdown
Member Author

@codex review

@mashkovd

mashkovd commented May 4, 2026

Copy link
Copy Markdown
Member Author

@copilot review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI commented May 5, 2026

Copy link
Copy Markdown

@copilot review

Reviewed the changes in a3263b0. Here's the full assessment:

Patch A — command/args lock-cleanup wrapper ✅
Both labs and ovk use the correct sh -c form. The find ... -exec rm -rf {} + (batch form) is correct, and exec node ... properly replaces the shell process so SIGTERM is forwarded directly to Node. No-op on a clean emptyDir.

Patch B1 — S3 plugin-runtime-deps purge ✅
Tenant-specific paths are correct: s3/platform-state/labs/openclaw/plugin-runtime-deps vs s3/platform-state/ovk/openclaw/plugin-runtime-deps. Correctly placed before the mc mirror call so the purged dir is never re-hydrated.

Patch B2 — post-restore lock cleanup ✅
find /home/node/.openclaw -type d -name '*.lock' -exec rm -rf {} + runs after mc mirror in the init container, cleaning any S3-poisoned owner.json-bearing lock dirs. The main container's Patch A wrapper also runs the same cleanup at start time — redundant but provides defence-in-depth.

Patches C/D — probe budgets ✅
Startup: timeoutSeconds: 10, failureThreshold: 90 → 455s budget (was 185s). Readiness: timeoutSeconds: 10, failureThreshold: 5. Liveness: timeoutSeconds: 15, failureThreshold: 10 (~5 min tolerance). Both tenants are consistent with admins and with each other.

Patch E — Bonjour disabled ✅
"plugins": { "entries": { "bonjour": { "enabled": false } } } is correctly placed in the JSON config in both labs and ovk.

Scope ✅
Image tag untouched (2026.4.26-beta.1), tenant-specific settings (CPU limits, env vars, envFrom, MCP servers, backstage block) unmodified. LGTM — no issues found.

@mashkovd
mashkovd merged commit 1bbf179 into main May 5, 2026
@mashkovd
mashkovd deleted the fix/labs-ovk-backport-admins-durability-clean branch May 5, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants