Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@ Versions follow [SemVer](https://semver.org).

### Added

- `OUTERLOOP_AUTO_UPDATE` (`off` by default, `release`, `main`) sets what the
deploy step does to a checkout-based deployment: nothing, move to the newest
release tag, or follow every merge. Adopters run what they installed until
they choose to upgrade; following `main` is for the kernel's own developers.
`init` writes the default for Slurm deployments. A successful sync records
the installed commit, so a checkout moved by hand rolls back the same way a
policy move does when its sync fails.
- A seed cache per target (docs/design/eval-cache.md): the tick warms one uv
cache from the target's lockfile on the tick host — wheels only, `--no-build`,
never the target's code — and every eval and launch job copies it into its
Expand Down Expand Up @@ -86,6 +93,8 @@ Versions follow [SemVer](https://semver.org).

### Changed

- The deploy step no longer pulls `main` on its own: a deployment that relied
on the PAT-gated pull sets `OUTERLOOP_AUTO_UPDATE=main` in its `.env`.
- Launches always queue. The two admission schemes tried this cycle — held
launches released under `OUTERLOOP_MAX_LAUNCH_GPUS`, then refusing a sleep's
launches while the account waited on a cap — are gone: the kernel keeps no
Expand Down
2 changes: 1 addition & 1 deletion docs/design/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ outage notice, not a round — because silence only visible in the
Actions tab is not a defensible way to miss a verification.

**Every flight has a snapshot.** Submitted jobs do not run from the
shared checkout — the deploy step resets it at every tick, so a queued
shared checkout — the deploy step may move it at any tick, so a queued
job could have its code swapped mid-flight. Each submission gets a
detached worktree of the checkout as it was at submit time (`flights/`,
beside the checkout); the job runs exactly the tree that submitted it,
Expand Down
11 changes: 11 additions & 0 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,17 @@ that can reach GitHub and your LLM provider works.
you start it again. On a headless machine start it under `nohup` or in a
tmux window, or as a user service.

**Updates.** A Slurm deployment runs from a checkout, and
`OUTERLOOP_AUTO_UPDATE` in `.env` says whether the deploy step moves it. `off`,
the default, leaves the checkout alone: you upgrade when you choose, and the
loop keeps running the code you validated. `release` moves it to the newest
release tag at the next cadence (pre-releases included; the repo is public, so
no credential is needed). `main` follows every merge and is meant for the
kernel's own developers. Whatever moves the checkout, you or the policy, the
environment is synced to the commit that is checked out, or the deploy rolls
back to the last commit whose environment was installed. The local loop runs the installed package: upgrade it with
`pip install -U outerloop-science` and start it again.

Experiments run wherever your `compute` backend says. Slurm is the first
backend; the interface is small (submit a job, poll for completion), so a CI
runner, a cloud backend, or a hardware rig plugs in the same way.
Expand Down
7 changes: 3 additions & 4 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,9 @@ standing instruction they supersede — a resumed agent honors stale constraints
`--dependency=singleton`, absolute `--begin` cadence grid, sbatch
retry/backoff; successors submitted FIRST so nothing below can break
the chain
- [x] Deploy shim (same script): pull main with the bot PAT → `uv sync
--locked` → exec tick; every step best-effort (bad merges crash ticks,
never the chain). Bot PAT on Torch + repo in the token's selection
still owed by the maintainer before live deploy
- [x] Deploy shim (same script): move the checkout per `OUTERLOOP_AUTO_UPDATE`
(`off` by default, `release`, `main`) → `uv sync --locked` → exec tick;
every step best-effort (bad merges crash ticks, never the chain)
- [x] Pause sentinel, heartbeat at tick start (now with disk preflight
payload), stale-lease reaping (TTL + tombstone CAS)
- [x] `compute`: sbatch submit / sacct status / cancel behind an injectable
Expand Down
2 changes: 1 addition & 1 deletion scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ runs the local loop elsewhere, reading placement from `~/.config/outerloop/.env`
| --- | --- |
| `setup_branch_protection.sh` | Apply/refresh branch protection on `main`. Solo-phase default: 0 approvals, checks required, admins enforced. Re-run with `1` once a second code owner joins. |
| `tick_chain.sbatch` | The Slurm tick entry: per-cadence chain (deploy, one tick, schedule the next) or, with `OUTERLOOP_RESIDENT=1`, the resident loop (`docs/design/resident-tick.md`). |
| `tick_deploy.sh` | The deploy step both modes source: stale-lock sweep, pull `main`, sync deps, read the operator's `.env` knobs, install backends. |
| `tick_deploy.sh` | The deploy step both modes source: stale-lock sweep, move the checkout per `OUTERLOOP_AUTO_UPDATE` (`off` by default; `release`, `main`), sync deps, read the operator's `.env` knobs, install backends. |
| `tick_resident.sh` | The resident loop: deploy → one tick under a timeout → sleep to the slot, one `afterany:self` successor, pause exits clean, shim changes resubmit the successor. |
| `requeue_moved_successors.sh` | Cancel pending same-name jobs that are no longer on the requested partition. The tick chain runs this before adding successors. |
| `sweep_git_locks.sh` | Remove stale `.git/*.lock` files from a checkout (older than N minutes, no live git process); the chain runs it before each deploy. |
Expand Down
11 changes: 6 additions & 5 deletions scripts/tick_chain.sbatch
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@
#
# Order matters and is the fail-safety:
# 1. top up successors FIRST — a crash anywhere below never breaks the chain
# 2. pull main + sync (deploy-at-tick-start; failures logged, tick continues
# on the previous code — a bad merge crashes ticks, never the chain)
# 2. update the checkout per OUTERLOOP_AUTO_UPDATE + sync (deploy-at-tick-
# start; failures logged, tick continues on the previous code — a bad
# merge crashes ticks, never the chain)
# 3. exec the tick
#
# Deploy/config knobs come from the environment (set once via
Expand All @@ -14,8 +15,8 @@
# OUTERLOOP_ACCOUNT / OUTERLOOP_PARTITION slurm placement (optional;
# unset uses Slurm's default association/partition)
# OUTERLOOP_CADENCE_MIN tick cadence, default 30
# OUTERLOOP_PAT_FILE bot PAT for the deploy pull (optional; no file
# means run whatever code is already deployed)
# OUTERLOOP_PAT_FILE bot PAT; a deploy fetch authenticates with it when
# present (optional — the kernel repo is public)
# OUTERLOOP_RESIDENT=1 run as the RESIDENT tick (design/resident-tick.md):
# one long-lived job that loops deploy -> tick ->
# sleep, with a single afterany:self successor.
Expand Down Expand Up @@ -135,7 +136,7 @@ if [ -w "$LOG_DIR" ]; then
fi
echo "=== tick $(date -Is) on $(hostname -s) job=${SLURM_JOB_ID:-none}"

# --- 2. deploy: pull main, sync deps, read the operator's knobs (best-effort;
# --- 2. deploy: update the checkout, sync deps, read the operator's knobs (best-effort;
# scripts/tick_deploy.sh, sourced so its exports reach the tick) ---
. "$OUTERLOOP_HOME/scripts/tick_deploy.sh"

Expand Down
Loading
Loading