Skip to content

Service runtime 0.0.42 cannot be started by a pre-0.0.42 service-launcher (layout changed to Node SEA; launcher not replaced) #12020

Closed as not planned
Closed as not planned
Service runtime 0.0.42 cannot be started by a pre-0.0.42 service-launcher (layout changed to Node SEA; launcher not replaced)#12020
Labels
duplicateThis issue or pull request already exists

Description

@john-e-west
john-e-west
opened on Sep 16, 2026

Summary

Upgrading the background-service runtime to 0.0.42 leaves the service
permanently unable to start. 0.0.42 replaced the JS bundle entrypoint with a
compiled Node SEA binary, but ~/.t3/runtime/service-launcher.mjs is not
replaced by the update and still resolves the old bundle path. The launcher
reports the new runtime as corrupt, systemd restarts it until the start limit is
hit, and the service ends in failed.

The runtime itself is fine. Only the launcher's path assumption is stale.

Impact

On this host the service went from healthy to permanently down with no user
action, and stayed down:

Active: failed (Result: start-limit-hit)
NRestarts=5
:3773 unreachable

Because Restart=always with StartLimitBurst=5, the unit exhausts its budget
in ~5 minutes and then requires a manual systemctl --user reset-failed. It does
not recover on its own. Any deployment that takes 0.0.42 with a pre-0.0.42
launcher on disk will hit this.

Root cause

service-launcher.mjs hardcodes the bundle entrypoint:

const runtimePaths = (baseDir, version) => {
	const versionDir = NodePath.join(baseDir, "runtime", "versions", version);
	return {
		versionDir,
		entryPath: NodePath.join(versionDir, "node_modules", "t3", "dist", "bin.mjs"),
		sentinelPath: NodePath.join(versionDir, ".install-complete")
	};
};

runtimeExists() stats entryPath, fails, and #startChild() throws
Selected t3@${version} runtime is missing or incomplete.

The two layouts differ:

0.0.40 0.0.42
entrypoint node_modules/t3/dist/bin.mjs (8.8 MB JS) ./t3 (compiled ELF, Node SEA)
top level node_modules, package.json, package-lock.json t3, client, resource-monitor, node_modules
node_modules holds the t3 package native addons only (ffi-rs, msgpackr-extract, @ff-labs, detect-libc)

0.0.42/.install-complete correctly contains 0.0.42, and the install is
complete — the sentinel check would pass if the entry path resolved.

The update ships no replacement launcher: nothing matching *launcher*
exists anywhere under versions/0.0.42/.

Reproduction

  1. Run the background service on 0.0.40 (or any bundle-layout release).
  2. Let it update its runtime to 0.0.42.
  3. Service fails to start.
$ journalctl --user -u t3code.service
Starting T3 Code server...
t3code.service: Main process exited, code=exited, status=1/FAILURE
...
$ tail ~/.t3/userdata/logs/boot-service.log
[service-launcher] Selected t3@0.0.42 runtime is missing or incomplete.

Suggested fix

Detect the layout rather than assume it, so one launcher starts either
generation and a rollback to a bundle release still works:

const runtimePaths = (baseDir, version) => {
	const versionDir = NodePath.join(baseDir, "runtime", "versions", version);
	const nativeEntry = NodePath.join(versionDir, "t3");
	const bundleEntry = NodePath.join(versionDir, "node_modules", "t3", "dist", "bin.mjs");
	let native = false;
	try {
		native = NodeFS.statSync(nativeEntry).isFile();
	} catch {
		native = false;
	}
	return {
		versionDir,
		native,
		entryPath: native ? nativeEntry : bundleEntry,
		sentinelPath: NodePath.join(versionDir, ".install-complete")
	};
};

and at the spawn in #startChild():

const spawnCommand = paths.native ? paths.entryPath : process.execPath;
const spawnArgs    = paths.native ? ["serve"] : [paths.entryPath, "serve"];
const child = NodeChildProcess.spawn(spawnCommand, spawnArgs, { /* unchanged */ });

The "ipc" stdio channel needs no change — the 0.0.42 binary is a Node SEA
(NODE_CHANNEL_FD and node:internal/child_process are present in the image),
so it still speaks the Node IPC protocol, and ./t3 --help shows it accepts
serve.

This fix is written but not yet verified in situ on this host — applying it
requires modifying a live service file and I have not done so. The analysis
above is verified; the patch is not yet proven by a successful start.

Two adjacent notes

A user-side wrapper has the same assumption. Any script that resolves the
active runtime by reading runtime/service-state.json and joining
node_modules/t3/dist/bin.mjs breaks identically. If that pattern is documented
anywhere, it needs the same detection.

Consider making the launcher self-updating or version-gated. The underlying
hazard is that the launcher and the runtime can drift, and the launcher is the
component that cannot be fixed by the update mechanism it supervises. A
minLauncherProtocol field in the runtime manifest — refusing to select a
runtime the on-disk launcher is too old to start, rather than selecting it and
failing — would turn this class of failure into a clean refusal plus a stay on
the previous version.

Environment

t3code-bin      0.0.38-1 (Arch/AUR)
runtimes        0.0.40 (working), 0.0.42 (cannot be started)
launcher mtime  2026-09-12, i.e. the 0.0.40-era file
node            26.8.1 via mise
OS              Arch Linux (Omarchy), kernel 7.1.9, systemd 261
unit            user service, Restart=always, StartLimitBurst=5/300s

Workaround

Roll the active runtime back and reset the failed unit:

printf '{\n  "protocol": 2,\n  "activeVersion": "0.0.40"\n}\n' > ~/.t3/runtime/service-state.json
systemctl --user reset-failed t3code.service
systemctl --user restart t3code.service

Activity

  1. juliusmarminge commented on Sep 16, 2026

    @juliusmarminge
    juliusmarminge
    on Sep 16, 2026
    Member

    Closing as a duplicate of #11934 — same root cause: a pre-#11510 / pre-SEA service-launcher.mjs still resolves node_modules/t3/dist/bin.mjs, so a complete standalone runtime (./t3) looks “missing or incomplete,” and the unit can hit start-limit-hit.

    That was fixed on main by #11940 (bump SERVICE_LAUNCHER_PROTOCOL to 3 so staged preflight blocks activation with the actionable “update the launcher on the server machine” message). Tag v0.0.42 points at 719a76ca (2026-09-15 ~21:29Z), which is before #11940 merged (~22:21Z), so stable 0.0.42 does not include that gate yet.

    Workaround (same as #11934 / your rollback): stay on / roll back to a bundle-layout runtime the on-disk launcher can start, or rewrite the launcher/unit with a current local t3 update from a new enough binary, then systemctl --user reset-failed + restart.

  2. juliusmarminge
    added
    duplicateThis issue or pull request already exists
    on Sep 16, 2026

  3. juliusmarminge commented on Sep 16, 2026

    @juliusmarminge
    juliusmarminge
    on Sep 16, 2026
    Member

    Duplicate of #11934 (fixed on main by #11940; v0.0.42 predates that merge).

  4. juliusmarminge
    closed this as not plannedon Sep 16, 2026

  5. john-e-west commented on Sep 16, 2026

    @john-e-west
    john-e-west
    on Sep 16, 2026
    Author

    Retracting this — the fault is local to my host, not t3code. Sorry for the noise.

    I filed this against the wrong component. The 0.0.42 update behaved
    correctly.     On closer inspection of the systemd unit:

    0.0.42 installed     2026-09-15 22:09:23
base unit rewritten  2026-09-15 22:09:25   <- two seconds later, by the update

    The updater did rewrite the service unit for the new layout:

    ExecStart=/home/jwest/.t3/runtime/versions/0.0.42/t3 __service-launcher

    and the shipped binary does accept __service-launcher. Nothing was stranded.

    What actually broke was a local systemd drop-in on my host, written on
    12 September to bind the server to the LAN, which also overrode the launch
    command:

    # ~/.config/systemd/user/t3code.service.d/lan-server.conf
ExecStart=
ExecStart=/usr/.../node /home/jwest/.t3/runtime/service-launcher.mjs

    ExecStart= clears the base unit's value, so my three-day-old override pinned
    the old node + service-launcher.mjs invocation and defeated the update. The
    failing process in the journal is that override, not anything t3code installed.

    So the premise of this report — "the update ships no replacement launcher" — is
    false. It ships a rewritten unit; my drop-in discarded it.

    My error, and worth naming precisely: I read the drop-in early in the
    investigation, read the base unit only after the update had rewritten it, and
    never re-read the effective ExecStart that systemd was actually running. The
    evidence that would have caught it — systemctl show -p ExecStart — was one
    command away throughout.

    The one suggestion I would still make, weakly

    A local ExecStart= override silently defeating a unit rewrite is easy to do
    and gives a confusing failure ("runtime is missing or incomplete" rather than
    "your unit is launching the wrong thing"). If the launcher can cheaply detect
    that it was started against a runtime whose layout it cannot read, a message
    naming the effective command and pointing at drop-ins would have saved me
    several hours. Entirely your call whether that is worth the code.

    Closing. Apologies for the misdirected report.

  6. john-e-west commented on Sep 16, 2026

    @john-e-west
    john-e-west
    on Sep 16, 2026
    Author

    Correcting my own retraction above — I over-corrected, and @juliusmarminge is right.

    Both things are true, and they compose:

    • [Bug]: background service self-update fails permanently against a pre-#11510 service-launcher.mjs #11934 is genuine: a pre-SEA service-launcher.mjs resolves
      node_modules/t3/dist/bin.mjs, so a complete ./t3 runtime reads as
      "missing or incomplete" and the unit reaches start-limit-hit. Exactly what
      happened here.
    • What put my host on that path is a local drop-in with ExecStart= +
      ExecStart=node .../service-launcher.mjs, written before the SEA change. The
      0.0.42 update did rewrite the base unit to
      ExecStart=.../0.0.42/t3 __service-launcher; my override cleared it and kept
      invoking the old launcher.

    So the upstream defect is real and my drop-in is the local trigger for it. My
    retraction said "not t3code", which was wrong — please disregard that line.
    A host with no ExecStart override would take the rewritten unit and never
    reach the stale launcher.

    Nothing needed here; #11934 / #11940 cover it. Noted for anyone who lands on
    this issue: if you hit this on a release predating #11940, check
    systemctl --user show t3code.service -p ExecStart for a drop-in override
    before assuming the update failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    duplicateThis issue or pull request already exists

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions