Skip to content

Disaster recovery: back up all data off Railway and GitHub, and a template that redeploys the whole project on a new account #8675

Open

Description

@gHashTag

The owner, 2026-10-10 (translated): "they may cut us off soon! If we move to another account, we must be able to redeploy all servers fast. Back up all the data and make a template for deploying the whole project quickly. Also find what is duplicated, what can be merged, and how to pay less for services."

At risk: the Railway and GitHub accounts (owner's answer). Backups go to cloud storage outside Railway (owner's choice): an S3-compatible bucket such as Cloudflare R2 or Backblaze B2, on a separate account.

What exists (inventory 2026-10-10, no secret values read)

  • Railway: 14 projects. Project 999 alone has 37 services in production.

  • Data that must survive:

    • Postgres (6.3 GB);
    • Postgres-NFrq (1.4 GB);
    • pgvector (2.2 GB);
    • Redis (1.5 GB);
    • the minio Bucket (12.7 GB);
    • the volumes of queen-watch (0.8 GB), skills-site (0.8 GB), telegram-bot-api (0.8 GB) and the browser-pod Chromium profile (1.9 GB, logins).

    The other volumes hold caches or re-downloadable models: check-runner 17 GB, the trios workspaces, vision, whisper.

  • Services with no source in git. They were deployed with railway up from a local folder: gigaam, mouth, parakeet, voice-clone, voice-clone-qwen, browser-pod, jobsearch-cron, queen-proxy, skills-site, telegram-bot-api. If Railway goes, their code may go with it.

  • GitHub: 239 repos under gHashTag, 215 of them not forks, 16.4 GB in all. Issues, PRs and comments are the project's ledgers, and they exist only on GitHub.

Plan

Phase 0, now, needs nothing from the owner

  1. A machine-readable manifest of every Railway project. For each service it records:

    • source (repo and branch, or image), root dir, build and deploy config;
    • cron, replicas, domains;
    • volumes (mount and size);
    • variable NAMES only.

    It is generated, not hand-typed, and a copy is kept off GitHub.

  2. Find the source of every CLI-deployed service. Push it to a git repo, or record the image digest, so nothing exists only inside Railway.

  3. A cost and duplication review. Merge candidates: three Postgres instances, three speech-to-text services (whisper, parakeet, gigaam), two voice-clone services, three browser services, the two igla-judge projects, finished *-once functions, and projects with generated names. Each recommendation names what it saves and what it risks.

Phase 1, needs the owner's bucket and keys, entered by the owner into Railway shared variables (never handled by an agent)
4. Nightly dumps to the bucket, using existing images with no hand-written backup code:

  • pg_dump of the three Postgres instances, for example the official postgres-s3-backups template;
  • a Redis RDB snapshot;
  • mc mirror of minio;
  • tar of the four data volumes.

Each dump is checked by restoring it into a scratch database.
5. GitHub off-site:

  • nightly git bundle --all of all 215 non-fork repos;
  • a JSON export of issues, PRs and comments for t27, trios, trinity and 999-multibots-telegraf;

all of it into the same bucket.
6. Variable values. The owner exports them with one command, encrypted with a passphrase only the owner knows. Agents never see the values.

Phase 2: the restore template
7. The deployment described in this project's own language: a .t27 topology spec covering services, sources, volumes, restore order and data versus cache, with tests. A generated restore plan for a new Railway account then replays it: create the services, attach the sources, create the volumes, load the variables from the encrypted file, restore the data.
8. A rehearsal into an empty scratch project, timed, before anyone needs it.

Owner actions are listed in the comments as they come up.

User Scenarios

  • Given the Railway account is accessible and the owner has provided an S3-compatible bucket with credentials in Railway shared variables, When the nightly backup runs, Then all three Postgres databases, Redis, MinIO bucket, and four data volumes are dumped to the bucket and each dump is verified by restoring into a scratch database.
  • Given the GitHub account is accessible and the owner has provided an S3-compatible bucket, When the nightly GitHub backup runs, Then git bundle --all archives for all 215 non-fork repos and JSON exports of issues, PRs, and comments for t27, trios, trinity, and 999-multibots-telegraf are stored in the bucket.
  • Given a fresh Railway account with no services, the encrypted variable file, and the bucket containing the latest backups, When the restore template is applied, Then all services are created with correct sources, volumes are provisioned, variables are loaded from the encrypted file, and data is restored in the correct order, passing the topology's tests.

Requirements

FR-001: The system MUST generate a machine-readable manifest of all 14 Railway projects with every service's source, build/deploy config, cron, replicas, domains, volumes, and variable names, and store a copy outside GitHub.
FR-002: The system MUST locate source code for all 10 CLI-deployed services (gigaam, mouth, parakeet, voice-clone, voice-clone-qwen, browser-pod, jobsearch-cron, queen-proxy, skills-site, telegram-bot-api) and push each to a Git repo or record its image digest.
FR-003: The system MUST produce a cost and duplication review naming merge candidates (three Postgres, three STT, two voice-clone, three browser, two igla-judge, finished *-once, generated-name projects) with savings and risks.
FR-004: Nightly backups MUST use existing images (e.g., postgres-s3-backups, mc mirror, tar) with no hand-written backup code, writing to the owner's S3-compatible bucket.
FR-005: Each nightly dump MUST be verified by restoring into a scratch database before the backup job succeeds.
FR-006: Nightly GitHub backups MUST create git bundle --all for all 215 non-fork repos and JSON exports of issues/PRs/comments for the four named projects into the same bucket.
FR-007: Variable values MUST be exported by the owner with a single command, encrypted with a passphrase only the owner knows; agents MUST never see plaintext values.
FR-008: A .t27 topology spec MUST describe the full deployment: services, sources, volumes, restore order, data vs. cache classification, and include tests.
FR-009: A generated restore plan MUST replay the topology on a new Railway account: create services, attach sources, create volumes, load variables from the encrypted file, restore data.
FR-010: A rehearsal restore into an empty scratch project MUST complete successfully and be timed before the template is considered ready.

Success Criteria

  • cargo run --bin railway-manifest -- --output railway-manifest.json produces a valid JSON manifest covering all 14 projects and 37+ services; jq empty railway-manifest.json exits 0.
  • ls specs/restore-topology.t27 exists and t27c validate specs/restore-topology.t27 exits 0.
  • t27c test specs/restore-topology.t27 exits 0 with at least 5 tests passing.
  • Nightly backup job railway backup run --project 999 --verify exits 0 and restores each of the three Postgres dumps, Redis RDB, MinIO mirror, and four volume tars into scratch services with data integrity checks passing.
  • git bundle verify bucket/gh-backup-$(date +%F).bundle exits 0 for all 215 repos; jq empty bucket/issues-t27-$(date +%F).json exits 0 for each of the four projects.
  • Restore rehearsal: t27c deploy specs/restore-topology.t27 --target scratch-restore --vars-file vars.enc --bucket bucket/latest exits 0 within 60 minutes and t27c test specs/restore-topology.t27 --target scratch-restore exits 0.

Boundary

  • specs/server/project.t27
  • specs/trinity/project.t27
  • specs/tri/utils/template.t27
  • specs/docs/chapters/project.t27
  • railway.toml
  • bootstrap/src/railway.rs
  • .trinity/seals/Account.json
  • .trinity/seals/Project.json
  • infra/t27c-lab/railway.json
  • docs/system/project.md

Activity

  1. gHashTag commented on Oct 10, 2026

    @gHashTag
    CollaboratorAuthor

    Phase 0 result: Railway inventory, lost sources, cost review (2026-10-10)

    Everything was read-only. I read variable names only, never values, and changed nothing on Railway. The full manifest (JSON) stays off GitHub in the coordinator's scratch directory.

    Counts

    • 14 projects. Each has a single production environment. There are 59 service instances; project 999 has 37 of them.
    • 26 volumes: 25 attached, plus 1 unattached old Postgres copy in woody. There are no Railway buckets.
    • Sources: 16 GitHub repos, 4 Railway template repos (minio), 14 images, 3 Railway functions, 20 CLI uploads, and 2 services with no deployment.
    • One cron (queen-watch, 0 * * * *). Every service runs 1 replica.
    • DATA volumes: 16, about 38.7 GB reported. Each 50 GB volume carries about 0.8 GB of filesystem overhead, so the real data is closer to 26 GB (estimate).
      • The epic's list misses 7 of them: woody Bucket (4.4 GB), woody Postgres, the unattached woody copy postgres-2026-08-29, project-brain Postgres and /paperclip, heartfelt-cat Postgres, and Leela /data.
    • CACHE: 8 volumes, 147 GB. MODEL: 2 volumes, 10.6 GB (whisper and vision).

    Lost sources (CLI uploads)

    Service Verdict Where
    gigaam FOUND IN GIT 999-multibots-telegraf deployment/gigaam at cfa6deab4; the blobs are identical on main
    parakeet FOUND IN GIT the same deployment/gigaam folder; only GIGAAM_MODEL/GIGAAM_QUANT differ
    mouth FOUND IN GIT deployment/mouth at cb0ff6183; the blobs are identical on main
    voice-clone FOUND IN GIT deployment/voice-clone at 96ff5136e2
    voice-clone-qwen FOUND IN GIT the same folder, deployed from the branch of the voice-clone PR (999-multibots-telegraf#3508) 20 min before the merge. It is a duplicate.
    browser-pod FOUND IN GIT deployment/remote-browser at 52c35d82c0. All 13 build steps match. The base neko/chromium:latest is unpinned.
    telegram-bot-api FOUND IN GIT deployment/telegram-bot-api at f003a8c659
    queen-proxy FOUND IN GIT trinity apps/queen-proxy at dfe63cdacc
    t27b-lab, t27b-lab-2 FOUND IN GIT t27 contrib/railway/t27b-lab
    t27c-lab FOUND IN GIT t27 infra/t27c-lab
    igla-judge board, loop FOUND IN GIT igla-coder-gpu (private) board/, remote/loop
    woody turbobaby-bot FOUND IN GIT gHashTag/turbobaby-user-bot
    remote-browser, browseros FOUND IN GIT an older deployment/remote-browser; BrowserOS (its build failed)
    skills-site NOT FOUND: exists only in Railway Node app skills-site@1.0.0 (node index.js, skills.t27.ai, /data volume)
    jobsearch-cron NOT FOUND: exists only in Railway Node via Railpack (cron.t27.ai)
    project-brain NOT FOUND: the overlay exists only in Railway public image paperclipai/paperclip@sha256:d9fc5aa8... plus a local folder copied in
    agents-acceptance-once, crm-door-once, heartfelt-cat todo-api Railway functions: code was in Railway only extracted to the scratch directory; no secrets in them
    igla-judge judge (both projects), dayo64-xcheck never ran, or a finished job nothing to lose

    Cost

    • Previous period (09-09 to 10-09): $699.07, of which memory is $493.78 (71%) and project 999 is $613.47.
    • Run-rate: the last 7 days cost $339.67, which is about $1,456 per 30 days (estimate). The current period agrees: $57.70 in about 1.4 days.
    • Per-service figures are estimates from the usage API, priced at the unit rates of the bill.

    Merge/delete candidates, ranked by saving (estimates per 30 days; nothing was stopped):

    1. Delete voice-clone-qwen, about $82.
      • Same image as voice-clone. It used 1 vCPU-min in 7 days and had 0 requests in 24 h.
      • vibee-render calls voice-clone.railway.internal by default.
    2. Retire zep + zep-nlp + pgvector, about $66.
      • zep-memory.ts calls Zep "a mirror, not a dependency"; zep-nlp alone holds 6 GB of RAM.
      • Risk: the brief loses Zep summaries. Dump pgvector first.
    3. Delete agents-acceptance-once and crm-door-once, about $39.
      • They are finished one-shots, yet each burns about 1 vCPU around the clock (about 10k vCPU-min in 7 days).
    4. STT/TTS, $12 to $49.
      • gigaam, parakeet and mouth used 1-3 vCPU-min each in 7 days while holding 1.2-2.4 GB of RAM.
      • Option one: merge gigaam and parakeet into one pod (same image). Option two: drop all three and use whisper, which voice.ts already falls back to.
      • Risk: a slower first sound and Russian STT quality.
    5. Delete project remote-browser, about $7. It is an older copy of browser-pod with no volume and no traffic.
    6. Merge into one Postgres, about $7 plus two fewer backup jobs. The NFrq clients must be identified before migrating (pg_stat_activity, Phase 1).
    7. Small items, about $2 together. heartfelt-cat (a demo idle since June), overflowing-renewal (apps/queen-web, no references found), the two minio Consoles, dayo64-xcheck, igla-judge ec9020f4 (empty), browseros (failed build).
    8. trios-railway is deployed twice (a 999 service and its own project). Keep the project copy.
    9. Do not merge supabase-gateway, postgrest, supabase-storage or Bucket. The gateway served 4,635 requests in 24 h.

    Items 1-4 save about $236 per 30 days (estimate). The largest spend is not duplication:

    • the swarm and labs: bee runners, trios-agent-server, t27b-lab and t27c-lab, about $730;
    • vision, about $155, with 15 GB of RAM.

    Restore risks found

    • inngest runs ghcr.io/ghashtag/inngest:mcp-auth, which is hosted by the GitHub account at risk.
    • These images are unpinned :latest: zep, zep-nlp, ankane/pgvector, llama.cpp:server and neko chromium. Their digests are recorded in the manifest.
    • trinity/.railway/railway.ts and apps/queen-proxy/RESTORE.md still describe queen-ollama/queen-bees, which no longer exist.
  2. gHashTag commented on Oct 10, 2026

    @gHashTag
    CollaboratorAuthor

    Stopped on the owner's approval, 2026-10-10 ~16:15Z. I ran railway down on each service, which removes its active deployment. The service config, variables and volumes stay. Nothing was deleted. Any of them comes back with a redeploy from the dashboard, or a push for repo-backed services. Permanent deletion waits 24–48 h of observation and, where there is data, a dump in the off-site bucket.

    Project Service 30-day run-rate (estimate)
    999 voice-clone-qwen (a duplicate of voice-clone) $81.58
    999 zep-nlp $59.71
    999 crm-door-once $19.86
    999 agents-acceptance-once $19.48
    remote-browser remote-browser (an old copy of browser-pod) $7.07
    999 zep $1.45
    overflowing-renewal trinity $0.68
    999 Console (minio UI) $0.42
    woody Console (minio UI) $0.30
    999 trios-railway (its own project keeps the live copy) $0.02

    That is about $190 per 30 days at the current rate. Every stop was verified as REMOVED. After the stops, app.t27.ai and api.t27.ai/health answer 200.

    Kept for now, waiting on a dump:

    • pgvector (999), whose data is Zep's memory;
    • heartfelt-cat (todo-api and Postgres).

    browseros, dayo64-xcheck and the empty igla-judge ec9020f4 already had no active deployment.

    Sources that exist only inside Railway: skills-site (skills.t27.ai) and jobsearch-cron (cron.t27.ai). specs/crons/railway-jobsearch-cron.t27 is a catalog card, not the code. railway ssh answers "You do not have access to this resource" for this CLI login. The owner needs to run railway login again so the files can be copied out of the running containers. Copies leave out .env* and are scanned by gitleaks before they go to a private repo.

    Seen on the way:

    • Bad certificates. skills.t27.ai and cron.t27.ai fail TLS (curl exit 60). Their railway.app URLs answer 200, so the certificates for those custom domains are not valid.
    • Pending approval. vibee-render's latest deployment is in NEEDS_APPROVAL.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions