bug: linux-arm64 tarball missing @anush008/tokenizers-linux-arm64-gnu native binding #3951

Summary

The published superset-linux-arm64.tar.gz (cli-v0.2.1) crashes immediately on superset start:

Error: Cannot find module '@anush008/tokenizers-linux-arm64-gnu'
Require stack:
- /opt/superset/lib/node_modules/@anush008/tokenizers/index.js

#3921 fixed this for linux-x64 by shipping the tokenizers native; the linux-arm64 target was missed.

Reproduce

On any Linux ARM64 host (Hetzner CCX-ARM/Neoverse-N1, AWS Graviton, Raspberry Pi 4/5, Oracle Ampere):

mkdir -p /opt/superset && cd /opt/superset
curl -fsSL -o /tmp/s.tar.gz https://github.com/superset-sh/superset/releases/download/cli-v0.2.1/superset-linux-arm64.tar.gz
tar xzf /tmp/s.tar.gz -C /opt/superset
SUPERSET_API_KEY=sk_live_… /opt/superset/bin/superset start

→ MODULE_NOT_FOUND for @anush008/tokenizers-linux-arm64-gnu. Host service exits before binding a port.

Root cause (three layers)

1. Build pipeline (packages/cli/scripts/build-dist.ts, packages/host-service/build.ts): the linux-arm64 target doesn't list @anush008/tokenizers-linux-arm64-gnu. Same omission fix(cli): ship onnxruntime-node + tokenizers natives in linux-x64 tarball #3921 plugged for linux-x64.
2. @anush008/tokenizers@0.0.0 (the pinned version in bun.lock): its optionalDependencies lists only tokenizers-darwin-universal, tokenizers-linux-x64-gnu, tokenizers-win32-x64-msvc — no arm64. So even if the build script were updated, bun install couldn't resolve the missing package.
3. npm registry: @anush008/tokenizers-linux-arm64-gnu@0.0.0 does not exist on npm (only public versions 0.5.0 and 0.6.0 are published — confirmed via https://registry.npmjs.org/@anush008/tokenizers-linux-arm64-gnu).

So a complete fix requires all three layers.

Proposed fix

Layer 1 (this repo, ready as a diff):

diff --git a/packages/cli/scripts/build-dist.ts b/packages/cli/scripts/build-dist.ts
@@ -82,6 +82,7 @@ const TARGET_NATIVE_PACKAGES: Record<Target, string[]> = {
 	"linux-arm64": [
 		"@libsql/linux-arm64-gnu",
 		"@parcel/watcher-linux-arm64-glibc",
+		"@anush008/tokenizers-linux-arm64-gnu",
 	],
 };

diff --git a/packages/host-service/build.ts b/packages/host-service/build.ts
@@ -26,6 +26,7 @@ const result = await Bun.build({
 		"@anush008/tokenizers",
 		"@anush008/tokenizers-darwin-universal",
 		"@anush008/tokenizers-linux-x64-gnu",
+		"@anush008/tokenizers-linux-arm64-gnu",
 		"@anush008/tokenizers-win32-x64-msvc",
 	],
 });

I can open a PR with this diff as soon as layers 2+3 are unblocked. Holding it back for now since CI would fail today (bun install can't resolve the missing package).

Layer 2+3 (maintainer-side): publish @anush008/tokenizers-linux-arm64-gnu@0.0.0 to npm and republish @anush008/tokenizers@0.0.0 with that variant in optionalDependencies.

If 0.0.0 is intentionally a custom-pinned fork rather than maintained alongside the public stream, an alternative is to bump the dependency to ^0.6.0 (public release where all platform variants are properly published).

Workaround for current ARM64 users

Until the upstream fix lands, native binding can be shimmed by hand:

curl -fsSL -o /tmp/t.tgz https://registry.npmjs.org/@anush008/tokenizers-linux-arm64-gnu/-/tokenizers-linux-arm64-gnu-0.6.0.tgz
mkdir -p /opt/superset/lib/node_modules/@anush008/tokenizers-linux-arm64-gnu
tar xzf /tmp/t.tgz -C /opt/superset/lib/node_modules/@anush008/tokenizers-linux-arm64-gnu --strip-components=1

After this, superset start boots and the host registers with the relay. Verified on Hetzner CCX-ARM (Ubuntu 24.04, Neoverse-N1).

Impact

Anyone running the CLI on Hetzner ARM, AWS Graviton, Oracle Ampere, Raspberry Pi, or any aarch64 Linux host. Closely related to making the CLI usable for the headless/SSH workflow tracked in #1513.

Related

• fix(cli): ship onnxruntime-node + tokenizers natives in linux-x64 tarball #3921 — the analogous fix for linux-x64
• bug: superset auth login hangs silently on headless servers when xdg-open is missing #3949 — sibling headless-CLI bug (auth login URL never surfaced) with a PR (fix(cli): surface auth URL so headless logins can complete #3950) ready to merge
• [feat] use in ssh server #1513 — use in ssh server (the broader headless story)