Description
The owner, 2026-10-10 (translated): "they may cut us off soon! If we move to another account, we must be able to redeploy all servers fast. Back up all the data and make a template for deploying the whole project quickly. Also find what is duplicated, what can be merged, and how to pay less for services."
At risk: the Railway and GitHub accounts (owner's answer). Backups go to cloud storage outside Railway (owner's choice): an S3-compatible bucket such as Cloudflare R2 or Backblaze B2, on a separate account.
What exists (inventory 2026-10-10, no secret values read)
-
Railway: 14 projects. Project
999alone has 37 services inproduction. -
Data that must survive:
- Postgres (6.3 GB);
- Postgres-NFrq (1.4 GB);
- pgvector (2.2 GB);
- Redis (1.5 GB);
- the minio
Bucket(12.7 GB); - the volumes of queen-watch (0.8 GB), skills-site (0.8 GB), telegram-bot-api (0.8 GB) and the browser-pod Chromium profile (1.9 GB, logins).
The other volumes hold caches or re-downloadable models: check-runner 17 GB, the trios workspaces, vision, whisper.
-
Services with no source in git. They were deployed with
railway upfrom a local folder: gigaam, mouth, parakeet, voice-clone, voice-clone-qwen, browser-pod, jobsearch-cron, queen-proxy, skills-site, telegram-bot-api. If Railway goes, their code may go with it. -
GitHub: 239 repos under gHashTag, 215 of them not forks, 16.4 GB in all. Issues, PRs and comments are the project's ledgers, and they exist only on GitHub.
Plan
Phase 0, now, needs nothing from the owner
-
A machine-readable manifest of every Railway project. For each service it records:
- source (repo and branch, or image), root dir, build and deploy config;
- cron, replicas, domains;
- volumes (mount and size);
- variable NAMES only.
It is generated, not hand-typed, and a copy is kept off GitHub.
-
Find the source of every CLI-deployed service. Push it to a git repo, or record the image digest, so nothing exists only inside Railway.
-
A cost and duplication review. Merge candidates: three Postgres instances, three speech-to-text services (whisper, parakeet, gigaam), two voice-clone services, three browser services, the two
igla-judgeprojects, finished*-oncefunctions, and projects with generated names. Each recommendation names what it saves and what it risks.
Phase 1, needs the owner's bucket and keys, entered by the owner into Railway shared variables (never handled by an agent)
4. Nightly dumps to the bucket, using existing images with no hand-written backup code:
pg_dumpof the three Postgres instances, for example the officialpostgres-s3-backupstemplate;- a Redis RDB snapshot;
mc mirrorof minio;- tar of the four data volumes.
Each dump is checked by restoring it into a scratch database.
5. GitHub off-site:
- nightly
git bundle --allof all 215 non-fork repos; - a JSON export of issues, PRs and comments for t27, trios, trinity and 999-multibots-telegraf;
all of it into the same bucket.
6. Variable values. The owner exports them with one command, encrypted with a passphrase only the owner knows. Agents never see the values.
Phase 2: the restore template
7. The deployment described in this project's own language: a .t27 topology spec covering services, sources, volumes, restore order and data versus cache, with tests. A generated restore plan for a new Railway account then replays it: create the services, attach the sources, create the volumes, load the variables from the encrypted file, restore the data.
8. A rehearsal into an empty scratch project, timed, before anyone needs it.
Owner actions are listed in the comments as they come up.
User Scenarios
- Given the Railway account is accessible and the owner has provided an S3-compatible bucket with credentials in Railway shared variables, When the nightly backup runs, Then all three Postgres databases, Redis, MinIO bucket, and four data volumes are dumped to the bucket and each dump is verified by restoring into a scratch database.
- Given the GitHub account is accessible and the owner has provided an S3-compatible bucket, When the nightly GitHub backup runs, Then
git bundle --allarchives for all 215 non-fork repos and JSON exports of issues, PRs, and comments for t27, trios, trinity, and 999-multibots-telegraf are stored in the bucket. - Given a fresh Railway account with no services, the encrypted variable file, and the bucket containing the latest backups, When the restore template is applied, Then all services are created with correct sources, volumes are provisioned, variables are loaded from the encrypted file, and data is restored in the correct order, passing the topology's tests.
Requirements
FR-001: The system MUST generate a machine-readable manifest of all 14 Railway projects with every service's source, build/deploy config, cron, replicas, domains, volumes, and variable names, and store a copy outside GitHub.
FR-002: The system MUST locate source code for all 10 CLI-deployed services (gigaam, mouth, parakeet, voice-clone, voice-clone-qwen, browser-pod, jobsearch-cron, queen-proxy, skills-site, telegram-bot-api) and push each to a Git repo or record its image digest.
FR-003: The system MUST produce a cost and duplication review naming merge candidates (three Postgres, three STT, two voice-clone, three browser, two igla-judge, finished *-once, generated-name projects) with savings and risks.
FR-004: Nightly backups MUST use existing images (e.g., postgres-s3-backups, mc mirror, tar) with no hand-written backup code, writing to the owner's S3-compatible bucket.
FR-005: Each nightly dump MUST be verified by restoring into a scratch database before the backup job succeeds.
FR-006: Nightly GitHub backups MUST create git bundle --all for all 215 non-fork repos and JSON exports of issues/PRs/comments for the four named projects into the same bucket.
FR-007: Variable values MUST be exported by the owner with a single command, encrypted with a passphrase only the owner knows; agents MUST never see plaintext values.
FR-008: A .t27 topology spec MUST describe the full deployment: services, sources, volumes, restore order, data vs. cache classification, and include tests.
FR-009: A generated restore plan MUST replay the topology on a new Railway account: create services, attach sources, create volumes, load variables from the encrypted file, restore data.
FR-010: A rehearsal restore into an empty scratch project MUST complete successfully and be timed before the template is considered ready.
Success Criteria
cargo run --bin railway-manifest -- --output railway-manifest.jsonproduces a valid JSON manifest covering all 14 projects and 37+ services;jq empty railway-manifest.jsonexits 0.ls specs/restore-topology.t27exists andt27c validate specs/restore-topology.t27exits 0.t27c test specs/restore-topology.t27exits 0 with at least 5 tests passing.- Nightly backup job
railway backup run --project 999 --verifyexits 0 and restores each of the three Postgres dumps, Redis RDB, MinIO mirror, and four volume tars into scratch services with data integrity checks passing. git bundle verify bucket/gh-backup-$(date +%F).bundleexits 0 for all 215 repos;jq empty bucket/issues-t27-$(date +%F).jsonexits 0 for each of the four projects.- Restore rehearsal:
t27c deploy specs/restore-topology.t27 --target scratch-restore --vars-file vars.enc --bucket bucket/latestexits 0 within 60 minutes andt27c test specs/restore-topology.t27 --target scratch-restoreexits 0.
Boundary
- specs/server/project.t27
- specs/trinity/project.t27
- specs/tri/utils/template.t27
- specs/docs/chapters/project.t27
- railway.toml
- bootstrap/src/railway.rs
- .trinity/seals/Account.json
- .trinity/seals/Project.json
- infra/t27c-lab/railway.json
- docs/system/project.md
Activity
gHashTag commentedon Oct 10, 2026
Phase 0 result: Railway inventory, lost sources, cost review (2026-10-10)
Everything was read-only. I read variable names only, never values, and changed nothing on Railway. The full manifest (JSON) stays off GitHub in the coordinator's scratch directory.
Counts
productionenvironment. There are 59 service instances; project 999 has 37 of them.woody. There are no Railway buckets.queen-watch,0 * * * *). Every service runs 1 replica.Bucket(4.4 GB), woodyPostgres, the unattached woody copypostgres-2026-08-29, project-brainPostgresand/paperclip, heartfelt-catPostgres, and Leela/data.Lost sources (CLI uploads)
deployment/gigaamat cfa6deab4; the blobs are identical on maindeployment/gigaamfolder; onlyGIGAAM_MODEL/GIGAAM_QUANTdifferdeployment/mouthat cb0ff6183; the blobs are identical on maindeployment/voice-cloneat 96ff5136e2deployment/remote-browserat 52c35d82c0. All 13 build steps match. The baseneko/chromium:latestis unpinned.deployment/telegram-bot-apiat f003a8c659apps/queen-proxyat dfe63cdacccontrib/railway/t27b-labinfra/t27c-labboard/,remote/loopdeployment/remote-browser; BrowserOS (its build failed)skills-site@1.0.0(node index.js, skills.t27.ai,/datavolume)paperclipai/paperclip@sha256:d9fc5aa8...plus a local folder copied inCost
Merge/delete candidates, ranked by saving (estimates per 30 days; nothing was stopped):
voice-clone-qwen, about $82.voice-clone. It used 1 vCPU-min in 7 days and had 0 requests in 24 h.voice-clone.railway.internalby default.zep+zep-nlp+pgvector, about $66.zep-memory.tscalls Zep "a mirror, not a dependency";zep-nlpalone holds 6 GB of RAM.agents-acceptance-onceandcrm-door-once, about $39.whisper, which voice.ts already falls back to.remote-browser, about $7. It is an older copy of browser-pod with no volume and no traffic.apps/queen-web, no references found), the two minio Consoles, dayo64-xcheck, igla-judgeec9020f4(empty), browseros (failed build).Items 1-4 save about $236 per 30 days (estimate). The largest spend is not duplication:
vision, about $155, with 15 GB of RAM.Restore risks found
inngestrunsghcr.io/ghashtag/inngest:mcp-auth, which is hosted by the GitHub account at risk.:latest: zep, zep-nlp, ankane/pgvector, llama.cpp:server and neko chromium. Their digests are recorded in the manifest.trinity/.railway/railway.tsandapps/queen-proxy/RESTORE.mdstill describequeen-ollama/queen-bees, which no longer exist.gHashTag commentedon Oct 10, 2026
Stopped on the owner's approval, 2026-10-10 ~16:15Z. I ran
railway downon each service, which removes its active deployment. The service config, variables and volumes stay. Nothing was deleted. Any of them comes back with a redeploy from the dashboard, or a push for repo-backed services. Permanent deletion waits 24–48 h of observation and, where there is data, a dump in the off-site bucket.That is about $190 per 30 days at the current rate. Every stop was verified as REMOVED. After the stops,
app.t27.aiandapi.t27.ai/healthanswer 200.Kept for now, waiting on a dump:
pgvector(999), whose data is Zep's memory;heartfelt-cat(todo-api and Postgres).browseros, dayo64-xcheck and the empty igla-judge
ec9020f4already had no active deployment.Sources that exist only inside Railway:
skills-site(skills.t27.ai) andjobsearch-cron(cron.t27.ai).specs/crons/railway-jobsearch-cron.t27is a catalog card, not the code.railway sshanswers "You do not have access to this resource" for this CLI login. The owner needs to runrailway loginagain so the files can be copied out of the running containers. Copies leave out.env*and are scanned by gitleaks before they go to a private repo.Seen on the way:
skills.t27.aiandcron.t27.aifail TLS (curl exit 60). Their railway.app URLs answer 200, so the certificates for those custom domains are not valid.vibee-render's latest deployment is in NEEDS_APPROVAL.