Skip to content

test: add a launch smoke test for the built payload - #3

Closed
tylerdotai wants to merge 2 commits into
unfallenwill:mainfrom
tylerdotai:test/launch-smoke
Closed

tylerdotai wants to merge 2 commits into
unfallenwill:mainfrom
tylerdotai:test/launch-smoke

Conversation

@tylerdotai

Copy link
Copy Markdown

Problem

The offline suite in tests/run.sh asserts on files. Nothing in the repo
proves that a built payload actually starts.

That gap is not academic. A package can build cleanly and still be broken at
runtime:

  • a native addon that cannot be dlopen'd (wrong ELF class, missing
    libcua_driver_sdk.so, an addon built against a different Electron ABI),
  • a runtime resource that is not where the app looks for it,
  • a payload the app cannot locate at all.

None of that is visible to a build that only checks that files exist — and the
build already warns about this class of problem in its own comments:

upstream moves these in/out between releases and gates their loading on
app.isPackaged (patched to true in patch_linux.js), so a missing directory
is a SILENT feature loss, not a build failure.

Change

Add tests/smoke-launch.sh, which launches the built payload headlessly and
waits for the app's own startup-readiness signal, plus a CI step that runs
it against the staged payload.

The app logs its lifecycle timings once every startup subsystem has
initialised. outcome":"ready" with failed_count: 0 is that contract, so
the test asserts on the app's own report rather than on a proxy signal (a
window appearing, a process staying alive).

The script:

  • starts an Xvfb display and forces the Electron X11 backend, because
    Electron prefers Wayland and there is no compositor under Xvfb,
  • launches the payload with an isolated HOME/XDG_* tree, so the test
    neither touches a real profile nor collides with a running desktop session's
    single-instance lock,
  • waits for the readiness signal, and fails if the app exits first, times
    out, reports a failed startup subsystem, or dies after reporting ready,
  • cleans up by PID only — never by process name, so a running desktop
    session of the same app is never touched.

It is a separate script rather than a new case in tests/run.sh, because
that suite is explicitly "no network, no DMG, no GUI"; this one needs a
display and a real build.

Verification

Against a payload built by this repo's own install.sh from the official
macOS 3.1.4 x64 DMG (Electron 42.8.0):

[smoke] launching /tmp/mcl-payload/start.sh on :90 (timeout 120s)
[smoke] the app started and reported readiness
rc=0

Against a payload whose launcher exits immediately:

[smoke] the app exited before reporting ready
[smoke] FAIL: the app exited before reporting ready
rc=1

So the test fails on a broken payload and passes on a good one, in both
directions.

CI

build.yml gains the xvfb package and one step after "Build packages":

      - name: Launch smoke test
        run: ./tests/smoke-launch.sh build/payload
        env:
          MMX_SMOKE_REQUIRE: "1"

MMX_SMOKE_REQUIRE=1 makes a missing Xvfb a hard error in CI instead of a
skip, so the step cannot silently pass for the wrong reason. Locally the
script skips with a clear message when Xvfb is absent.

Notes

  • The script accepts either layout: this repo's start.sh, or a bare
    Electron binary with resources/ beside it (the upstream app layout), so
    it can also be pointed at a payload built another way.
  • Readiness took about 3 s on a warm machine, so this does not meaningfully
    slow the build.
  • No changes to the build pipeline itself; this only adds a check.

The offline suite asserts on files; nothing proved that a built payload
actually starts. A package can build cleanly and still be broken at
runtime -- a native addon that cannot be dlopen'd, a runtime resource
that is not where the app looks for it, an addon built against the
wrong Electron ABI -- and none of that is visible to a build that only
asserts on files.

tests/smoke-launch.sh launches the staged payload under Xvfb with an
isolated HOME/XDG tree and waits for the app's own startup-readiness
signal (outcome:"ready" with failed_count: 0), so the check is on the
app's report rather than on a proxy signal. It fails if the app exits
first, times out, reports a failed startup subsystem, or dies after
reporting ready.

build.yml runs it after "Build packages", with the xvfb package added
and MMX_SMOKE_REQUIRE=1 so a missing Xvfb is an error in CI rather
than a silent skip.

Verified against a payload built by this repo's install.sh from the
official macOS 3.1.4 x64 DMG: ready in ~3s, exit 0. Against a payload
whose launcher exits immediately: exit 1.
The launcher is started with 'cd $PAYLOAD' and an absolute path, but
$LAUNCHER was built from $PAYLOAD as given. With a relative payload
directory (which is how the CI calls it: ./tests/smoke-launch.sh
build/payload) the launcher resolved to build/payload/build/payload/start.sh
and the app never started.

Resolve $PAYLOAD to an absolute path first.
@tylerdotai

Copy link
Copy Markdown
Author

Closing this in favour of a single consolidated issue (#6) — four findings in one place, so there is one thing to read and reply to rather than three overlapping PRs. The work is not lost: the branch is still up in the fork and I will open a PR for any part of it you want.

@tylerdotai tylerdotai closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant