Skip to content

Rotate the t3fac log so it stops growing - #20202

Closed
rawphp wants to merge 259 commits into
pingdotgg:mainfrom
original-solutions:t3/factory-issue-60
Closed

rawphp wants to merge 259 commits into
pingdotgg:mainfrom
original-solutions:t3/factory-issue-60

Conversation

@rawphp

@rawphp rawphp commented Oct 16, 2026

Copy link
Copy Markdown

Closes #60

Contract

  • done: factory/scripts/rotate-log.sh <file> [max_mb=20] [keep=5] is a no-op under the limit; over it, it deletes anything numbered keep or higher, shifts .1….<keep-1> up, copies the file to .1, truncates it in place and gzips .2. A second LaunchAgent io.github.rawphp.t3fac.logrotate (template in factory/launchd/, StartInterval 600) runs it on t3fac.log. install.sh and rollback.sh copy the script to ~/.local/share/t3fac/rotate-log.sh, write the plist and reload that job before restarting the server; restart_job never touches it. README has a Logs section.
  • out: rotation inside the server, newsyslog, any upstream file, the main plist's log path.
  • risk: none
  • test: vp test run apps/server/src/factory/rotateLog.test.ts apps/server/src/factory/installScripts.test.ts → pass (50 tests)

Acceptance criteria

  • Script is a no-op under max_mb; over it copies to .1, shifts and gzips .2+, deletes past keep, truncates in place — rotateLog.test.ts: "does nothing under the limit", "keeps at most five rotated files over six runs", "removes everything past a smaller keep", "honours max_mb 0 and keep 1"
  • Second LaunchAgent with StartInterval 600, written and loaded by install.sh and rollback.sh like restart_job; restarting the main job does not touch it — installScripts.test.ts: "writes, installs and bootstraps the logrotate job on a first install", "boots the logrotate job out and back in on a second install, leaving the main restart alone", "rolls back to the previous version"
  • 21 MB file held open for append: next write lands at offset 0, .1 holds the old content, six runs keep at most five rotated files — rotateLog.test.ts: "copies to .1 and truncates in place while a writer holds the file open", "keeps at most five rotated files over six runs"
  • README names the log path, the limits and how to change them — verified factory/README.md § Logs

Decisions the issue left open

  • The release archive does not carry factory/scripts, and the checkout install.sh runs from can be a worktree that moves, so the scripts copy rotate-log.sh to ~/.local/share/t3fac/rotate-log.sh (atomic rename) and the plist runs that copy.
  • The logrotate job is reloaded immediately before restart_job, so the server restart stays the last launchd action before wait_healthy. Its reload is best effort: a failure prints warning: could not reload io.github.rawphp.t3fac.logrotate; log rotation is not running and the server restart, health check and rollback branch run as before (reliability review).
  • The rotator's own output goes to ~/Library/Logs/t3fac/logrotate.log; it prints one line per rotation and nothing otherwise.
  • O_APPEND check from the issue's notes: lsof +fg on two running LaunchAgents on this machine shows the AP flag on their StandardOutPath descriptors, so copytruncate is safe. The needs-info stop did not apply.

Gate

  • vp test run apps/server/src/factory → exit 0 (second run; the first run exited 1 on build.sh smoke test > fails and releases nothing when the server never answers, a pre-existing 1 s timing flake in a test this issue does not touch, see leftovers)
  • vp run --filter t3 typecheck → exit 0
  • Also: factory/scripts/check-seams.sh exit 0; plutil -lint on the rendered logrotate plist OK; no seam added.

Models

  • maker: claude-sonnet-5-5 (host; the repo has no models block, Tom's roster)
  • checker: claude-opus-5-5 (host)

Review

  • cycles: 2
  • cycle 1: correctness Approve (1 nit), reliability Request changes (1 action: a failing logrotate reload aborted install.sh/rollback.sh before the server restart). Both fixed in 27f05d8fb/256b3163d.
  • cycle 2: orchestrator read the fix diff; testing (mutating) review on 256b3163d: 12/12 mutants killed, 1 nit (stray temp file when the copy fails) fixed in dbc1e513c/40b0e2efe by the orchestrator.
  • leftovers: build.sh smoke test > fails and releases nothing when the server never answers can fail under full-suite load (T3FAC_SMOKE_TIMEOUT 1 s expires before the fake server writes its pid file). Pre-existing, build.sh and that test are outside this issue's files.

Lessons

  • Red-check worktrees need every package's node_modules, not just the root's. Symptom: tests in a detached worktree fail to collect with "Cannot find package 'effect/Context'" while the maker's tree is green. Cause: pnpm keeps a node_modules per workspace package, and only the root one was linked. Default: link every node_modules found with find . -maxdepth 3 -name node_modules (excluding nested ones) into the detached tree before running tests there.

Commits

  • e5a67b391 test(factory): rotate the t3fac log and install its LaunchAgent — Make slice (red: 13 failing)
  • c45494dbd feat(factory): rotate the t3fac log with a second LaunchAgent — Make slice
  • 27f05d8fb test(factory): a failing logrotate reload must not block recovery — Check cycle 1 (red)
  • 256b3163d fix(factory): a failing logrotate reload no longer blocks recovery — Check cycle 1
  • dbc1e513c test(factory): a failed script copy leaves no temporary file behind — Check cycle 2 (red)
  • 40b0e2efe fix(factory): remove the temporary script when its copy cannot land — Check cycle 2

Files

  • factory/scripts/rotate-log.sh — the copytruncate rotator
  • factory/launchd/io.github.rawphp.t3fac.logrotate.plist — the timer LaunchAgent template
  • factory/scripts/lib/t3fac.sh — reload_job, restart_logrotate_job, best-effort wrapper; restart_job unchanged in behaviour
  • factory/scripts/install.sh, factory/scripts/rollback.sh — reload the logrotate job before the server
  • factory/README.md — Logs section
  • apps/server/src/factory/rotateLog.test.ts — new
  • apps/server/src/factory/installScripts.test.ts — per-label launchctl stub, logrotate cases

Labels created this run: none (all seven existed).

rawphp and others added 30 commits October 15, 2026 20:00
…order-free

Red: "names an upstream file renamed into an allowed path" fails because
check-seams.sh lets git rename detection hide the deleted upstream path.
The fail case now changes server.ts itself, and the boundary real-tree
test asserts server.ts imports ./factory/index.ts; both already pass.

Issue: #1
A git mv of an upstream file into an allowed path listed only the new
path, so the deleted upstream file passed the seam check. --no-renames
lists both sides.

Issue: #1
- boundary.test.ts flags factory code outside factory/t3/ that imports the
  server's workspace packages (@t3tools/*, effect-acp, ...), read from
  apps/server/package.json.
- check-seams.sh captures the diff before the loop, so a failed git diff
  fails the script instead of passing with nothing checked.
- README and the script header say the check covers committed HEAD only.
- check-seams tests cover apps/web/src/factory/ and a failing diff.
- Node built-ins imported as NodeFS/NodePath/... per namespace-node-imports.
Factory skeleton: an empty FactoryLayer wired in through one seam
… plainly

Review nits: the cd to the repo root was never exercised, and the expected
calls rebuilt the constants they came from. No red phase: both behaviors
already held.

Issue: #2
Red on purpose: install.sh, rollback.sh and build.sh do not exist yet.

Issue: #3
… tail

Red: apps/server/src/factory/process.ts does not exist yet.

Issue: #8
…ests

Ports the run id, next_attempt, format_ts and parse_ts tests from softfac's
ledger.rs, plus the issue key parse and refuse cases from issue_key.rs.

Red: runId.test.ts and time.test.ts fail because runId.ts, time.ts and
errors.ts (LedgerError, used by testSupport.ts) do not exist yet.

Issue: #9
Ports softfac's RunId, IssueKey, next_attempt, format_ts and parse_ts to
plain TypeScript, and LedgerError with the Rust Display messages.

- Run ids are lowercase owner/name plus a canonical issue (up to u64 max) and
  attempt (up to u32 max); file names that are not exactly what this code
  writes are not run ids.
- Lowercasing is ASCII-only so a Kelvin sign cannot fold into a legal name.
- validateRunId also checks the issue and attempt, since they end up in the
  file name too.

Issue: #9
Ports the run ledger tests from softfac's ledger.rs: one exact line for every
one of the 30 event types, create/open semantics (create-new, newline repair
of a torn last line, symlinks and directories refused), the read rules (torn
last line, blank lines, invalid lines counted, unknown events and odd data
kept as written, causes kept as written), finish (total_ms and gate_attempts
computed by the writer), the failed-write rules through an injected AppendIo,
file and directory modes, and the directory-sync failure that removes the new
file. The feedback read file and the list/summary readers are not ported.

Red: ledger.test.ts fails because events.ts, appendFile.ts, read.ts and
ledger.ts do not exist yet.

Issue: #9
Ports softfac's find_project and status cases from src/t3.rs, adds launch
and thread cases, and a contract test over the real orchestration layers
with the Claude replay transcript. Red: paths.ts and sessions.ts do not
exist yet, so none of the three files compile.

Issue: #5
Red: the wrapper sh dies on SIGTERM and the group is SIGKILLed at once,
so the grandchild's marker never reaches the tail.

Issue: #8
Issue: #10

Both suites fail: ./lifecycle.ts and ./resultLine.ts do not exist yet.
Ports softfac's Ledger (create, open, append, finish) and read_run to plain
TypeScript over node:fs, writing the byte-for-byte line format
{"ts","ms","event","data","cause"}.

- create is create-new (mode 0600), makes directories 0700, syncs runs/, then
  the data dir and its parent when it made them, and removes the new file if a
  sync fails.
- One write plus one fsync per line; a short write is carried on, and after any
  failed write or sync the next line starts with a newline. A line counts as
  written once it is in the file, so a failed sync on finished cannot lead to a
  second finished line.
- open ends a torn last line with a newline; open and readRun refuse a symlink
  or directory named like a run.
- Event names are checked against the 30 in ledger.md and data fields are
  written in ledger.md order; finished gets total_ms and gate_attempts from the
  writer.
- readRun and parseLines follow the torn-last-line, blank-line and
  invalid-line rules and return data as written.

ledger.test.ts now carries a globalDate diagnostic suppression for the one
test that reads the system clock.

Issue: #9
…e probe

The cargo step now runs from / with its own CARGO_HOME under the gitignored target/, so a host-level crates.io source replacement in a parent or home cargo config no longer breaks the build.

Each smoke probe gets curl --max-time 2, and the poll loop breaks once 60s have elapsed, so a connection that hangs while the server initialises cannot stall the build.

Issue: #3
Covers the machine log from softfac's ledger.md and ledger.rs: the five
example lines as exact bytes with no cause, one exact line for each of the 17
event types and the attention and claim_skipped variants (the paused claim
line has only a reason), unlisted event names refused, a torn last line ended
on open, a symlink, directory or pipe at events.jsonl refused, a failed write
or sync followed by a line break, the directory fsyncs when the file is new,
and scanMachineLog (oldest first, blank and unparsable lines skipped, a last
line with no newline still read, a missing log is empty).

Red: machineLog.test.ts fails because machineLog.ts and the machine event
definitions in events.ts do not exist yet.

Issue: #9
Ports softfac's MachineLog and scan_machine_log: <data>/events.jsonl in the
ledger's line format without a cause.

- open makes the data dir (0700 when new) and the file (0600, O_NOFOLLOW),
  fsyncs the directories when the file is new, and ends a torn last line with a
  newline; a symlink, directory or pipe in its place is refused.
- append refuses event names ledger.md does not list and writes the data
  fields in ledger.md order, so the paused claim_skipped and the machine-level
  attention lines carry only their own fields.
- scanMachineLog returns the lines oldest first, skips blank and unparsable
  lines, still reads a last line with no newline, and opens without following
  a symlink or blocking on a pipe.

events.ts gains the 17 machine event types and their field order; index.ts
re-exports the public API.

Issue: #9
Issue: #10

U+0085 is stripped and U+FEFF kept, as Rust's White_Space set says and
JS trim does not. The comment on the 2^53 case now calls the divergence
from u64 what it is: deliberate and unreachable.
…er too, kill on interrupt

After SIGTERM, wait up to termGraceMs for the leader to exit and the
group to be empty before SIGKILL; in shell mode the wrapper sh died
first and the real command got no grace. Also signal the leader pid
alongside the group (softfac kill_all), and spawn inside
acquireUseRelease so an interrupt always kills the group and destroys
the pipe. The leader-left-group and interrupt cases have no
deterministic test.

Issue: #8
… line

Issue: #10

Pins the marker to the start of the line: a non-anchored match passed
every ported test, because softfac's list only prefixes FACTORY-STOP.
rawphp and others added 26 commits October 16, 2026 16:24
Adds the factory route layer under factory/t3. withScope authenticates
through EnvironmentAuth.authenticateHttpRequest, the call
AuthHttp.layerAuthenticatedAuth makes, then checks the route's scope.
FactoryLoop now carries a status (running, or idle with a reason) which
GET /api/factory/status reports. server.ts merges the routes in one line.
Question 1 in the design doc is answered under "Decided".

Issue: #57
attention.ts does not exist yet, so the whole file fails to load.

Issue: #58
Items are kept in attention.json, a new one posts one osascript notification with its
text in argv, and the third erroring pass in a row raises pass-errors.

Issue: #58
The fake world has an attention seam the pass does not use yet, so the six new tests fail.

Issue: #58
…across a restart

index.ts does not provide Attention or put it on the world yet, so these tests and the
tests that run a pass fail.

Issue: #58
The 200 cases hold only orchestration:read, a settings:write-only session
gets 403, and one request names /api/factory/status literally.

Issue: #57
Attention lines, stuck and needs-info runs raise items; a claim, a repo leaving the config
and a finished run clear them. The live world is wired in the next commit.

Issue: #58
The runtime provides Attention, raises a config item when the machine config does not load,
clears it when it does, applies notify, and reports how each pass ended.

Issue: #58
…notifying

Fails: a claim whose launch ends stuck leaves the issue's earlier stuck item
open beside the new one; a timed-out osascript is logged per item instead of
once and every later item runs it again; the osascript run carries no
termGraceMs.

Issue: #58
…il is 50 lines

Review findings: nothing exercised the "no factory config" line, and the
tail assertion could not tell 50 lines from the whole log. Both already pass.

Issue: #59
…otifying

A claim whose launch ends stuck now clears the issue's earlier items as a
started claim does. A timed-out osascript turns notifying off until restart
and is killed after a 1 s grace. A failed write or fsync of attention.json
no longer leaves its temp file behind.

Issue: #58
…tches build.sh

- http.test.ts: a route whose auth lookup fails inside T3 answers 500 internal_error and never
  runs its handler (review f2).
- Session tokens are issued in the test's Effect on the live clock instead of through
  Effect.runPromise, which the t3code lint rule forbids in tests.
- design.md step 6 describes the smoke test build.sh runs (environment 200, "factory started"
  in the log); it never called /api/factory/status, which needs a paired session (review f4).
…nfig notes

- rollback.sh restores previous when current is missing, naming only the restored version on failure
- tests: a server whose log line or port comes up after the first check is kept
- README: a factory config error fails the health check; hand-back moves config.yml aside when the repo is the only one
- plist and lib/t3fac.sh point at each other for the port and log path
…unning means

GET /api/factory/status goes to every client holding orchestration:read, so the idle reasons are
now the two the issue names, "no factory config" and "factory config failed to load"; the config
path and each config error stay in the server log, where they already went (review f1).

FactoryStatus says `running` means the pass loop started, not that passes succeed; pass health
is in the log until the status route grows (#27) (review f3).
Needs attention: keep a list of what needs a person and notify on the Mac
Recovery: roll back an install that does not come up, and a hand-back runbook
Factory CI: a Lint job runs upstream's lint and format checks
# Conflicts:
#	apps/server/src/factory/index.test.ts
#	apps/server/src/factory/index.ts
Factory route layer behind T3's authentication
Red: rotate-log.sh does not exist, so every rotateLog case fails (exit 127),
and installScripts expects the logrotate plist, the copied script and its
launchctl calls, none of which install.sh or rollback.sh produce yet. The
launchctl stub now tracks loaded and drain state per job label.

Issue: #60
rotate-log.sh copies the log to .1 and truncates it in place once it passes
max_mb (20), keeping 5 files (.2 and older gzipped). The logrotate LaunchAgent
runs it every 10 minutes from ~/.local/share/t3fac, so a crash-looping server
cannot grow the log without bound. install.sh and rollback.sh install and
reload it just before the server's restart; restarting the server alone does
not touch it.

Issue: #60
install.sh and rollback.sh must warn and carry on when the logrotate job
cannot be reloaded. Fails now: both exit 5 before restarting the server
(set -e aborts at restart_logrotate_job).

Issue: #60
restart_logrotate_job is best effort: install.sh and rollback.sh call it
through restart_logrotate_job_or_warn, which prints one warning to stderr
and carries on, so the server restart, health check and install rollback
branch still run. The function chains its steps with && so a failure
returns non-zero even though set -e is suppressed in a || context.

The installed rotate-log.sh is now copied to a temp file and renamed
over the old one, so a rotation running mid-install never reads a mix
of old and new text.

Issue: #60
Red: restart_logrotate_job leaves $root/.rotate-log.XXXXXX when the rename
over rotate-log.sh fails; the new install.sh case finds the stray file.

Issue: #60
@rawphp

rawphp commented Oct 16, 2026

Copy link
Copy Markdown
Author

Opened against the wrong repository by mistake; this belongs to the original-solutions/t3code-factory fork. Sorry for the noise.

@rawphp rawphp closed this Oct 16, 2026
@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants