Skip to content

Remote server update fails on Linux when node-pty must compile (no linux-x64 prebuild; opaque error) #6012

Closed
Closed
Remote server update fails on Linux when node-pty must compile (no linux-x64 prebuild; opaque error)#6012

Description

@MayberryDT
MayberryDT
opened on Aug 10, 2026

Summary

Remote/boot-service self-update can fail on Linux while preparing a pinned t3@<version> runtime because node-pty has no linux-x64 prebuild and the native rebuild fails. The desktop/UI only shows:

Server update failed: Could not prepare t3@0.0.33.

The underlying npm/node-gyp error is not surfaced, so operators have to dig through server npm debug logs and effect traces.

Environment

  • Host: Ubuntu 24.04 (Linux x64)
  • T3 server mode: boot-service / headless (t3 service, T3CODE_HOME pinned runtimes under runtime/versions/)
  • From version: 0.0.32
  • Target version: 0.0.33
  • Node: v22.23.1
  • System compiler: g++ → g++-13 (no g++-11 package installed)

What happens

  1. Client requests remote update to exact version 0.0.33.
  2. Server runs ensurePinnedRuntimeInstalled → npm install --prefix <staging> --no-fund --no-audit t3@0.0.33.
  3. Install pulls node-pty@1.1.0.
  4. node-pty install script: node scripts/prebuild.js || node-gyp rebuild.
  5. Prebuild check: no prebuilds/linux-x64 in the published package (only darwin/win32 prebuilds present).
  6. Rebuild fails. In our case the make step looked for g++-11 and exited 127.
  7. npm exits 1 → PinnedRuntimeInstallError → UI: Could not prepare t3@0.0.33.

Relevant packaging observation on node-pty@1.1.0 tarball contents: prebuilds exist for darwin-arm64, darwin-x64, win32-arm64, win32-x64 only.

Expected

  1. Linux installs should not require a local C++ toolchain for a normal server update (ship linux-x64 prebuilds for node-pty, or vendor a prebuilt binary for the supported Node ABI).
  2. If native compile still fails, the remote update error should include the actionable cause (e.g. missing compiler / node-gyp stderr tail), not only Could not prepare t3@X.

Workaround that unblocked us

# stage with explicit compiler
CXX=g++ CC=gcc npm install --prefix ~/.t3/runtime/versions/.staging-... --no-fund --no-audit t3@0.0.33
# then t3 service update / normal activation

# prevent recurrence on the user service
# ~/.config/systemd/user/t3code.service.d/40-native-build.conf
# [Service]
# Environment=CXX=g++
# Environment=CC=gcc

Why this matters

Headless / remote Linux boxes are a first-class T3 deployment path (t3 service, Tailscale, mobile/desktop remote control). Forcing an opaque native compile on every server version bump makes remote updates fragile and hard to support.

Nice-to-haves

  • Document remote-update prerequisites (Node version, compiler, disk) if native build remains required on some platforms.
  • Consider caching successful node-pty builds across staged versions on the same Node ABI to avoid rebuild thrash.

Happy to provide redacted npm debug log excerpts if useful.

Activity

  1. keithce commented on Aug 31, 2026

    @keithce
    keithce
    on Aug 31, 2026

    Confirmed another instance of this failure through t3 triage, with a different native-build trigger and an exact actionable cause.

    Environment

    • Server: t3code.service systemd user background service
    • Server version: t3@0.0.37-nightly.20260829.1219
    • Target: t3@0.0.38-nightly.20260831.1236
    • Server host: Linux x64, Node 24.20.0
    • Client: macOS nightly desktop app 0.0.38-nightly.20260831.1236
    • Connection: remote through T3 Connect

    Confirmed failure

    Three remote-update attempts failed identically. The UI only reported:

    Server update failed: Could not prepare t3@0.0.38-nightly.20260831.1236.

    The server trace recorded PinnedRuntimeInstallError with npm exit code 1. Matching npm debug logs showed node-pty@1.1.0 falling back to node-gyp rebuild, after which GCC failed with:

    fatal error: error writing to /tmp/cc*.s: Disk quota exceeded

    The actionable cause was a per-user quota on /tmp, not globally exhausted disk space:

    • /tmp: 15.7 GiB tmpfs mounted with usrquota
    • User-owned data in /tmp: approximately 12.588 GiB, apparently at the per-user limit
    • df still reported approximately 2.9 GiB globally available
    • /home had approximately 66 GiB free

    This is easy to misdiagnose if the updater checks only filesystem-wide free space.

    The current error path retains the npm exit code and stdout/stderr lengths in PinnedRuntimeInstallError, then exposes the generic Could not prepare t3@… wrapper. The useful npm stderr never reaches the remote client.

    Suggested improvements

    1. Surface a bounded, sanitized installation diagnostic. Known failures could be classified as temporary-directory quota exceeded, filesystem out of space, missing compiler/toolchain, or permission denied. Raw npm output should remain private because it may contain paths, registry configuration, or credentials.
    2. Preflight the temporary directory before starting a native build, accounting for per-user quota rather than only df/statfs free space.
    3. Consider running pinned-runtime installation with a private TMPDIR under the T3 state directory or another user-writable location with sufficient capacity.
    4. For unclassified errors, retain the redacted fallback but direct the operator to the relevant server/npm log.

    No workaround was applied during triage.

  2. cjkihl commented on Sep 9, 2026

    @cjkihl
    cjkihl
    on Sep 9, 2026

    Additional data point: same failure on linux-arm64 with t3@0.0.40 (stable) — so this isn't x64-only.

    • Host: Linux aarch64 (KVM VM, Ubuntu), Node 24.21.0
    • npx t3@latest service install succeeded, but t3code.service crash-looped with NodePtyModuleLoadError: Failed to load node-pty for linux-arm64. The bundled node-pty@1.1.0 ships only darwin-*/win32-* prebuilds — no linux-* at all.
    • Workaround that got the service green: npx node-gyp rebuild inside ~/.t3/runtime/versions/0.0.40/node_modules/node-pty, copy build/Release/pty.node → prebuilds/linux-arm64/pty.node, restart the service. (This box had make/g++/python3; without a toolchain this path is a dead end, which is the core of this issue.)
    • Caveat: the fix lives in the per-version runtime dir, so every service update needs it redone until node-pty ships Linux prebuilds (upstream Prebuilt binary for Linux ARM is actually x86-64 microsoft/node-pty#860 → PR UI Bugs when pinging files #857, 1.2.0 line).

    +1 for shipping Linux prebuilds (or vendoring the binary) and for surfacing the node-gyp stderr instead of Could not prepare t3@X.

  3. juliusmarminge commented on Sep 14, 2026

    @juliusmarminge
    juliusmarminge
    on Sep 14, 2026
    Member

    Fixed by the self-contained CLI / release-archive runtime stack that just landed on main (notably #11316 and #11510).

    Pinned runtimes are unpacked from signed release archives with prebuilt natives — no npm install / node-gyp compile path on remote or boot-service update anymore.

  4. juliusmarminge
    closed this as completedon Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions