Skip to content

Isolate test runs from every app-support store, plus the SP4 port spec and plan (SP4 batch A) - #20

Merged
donky-ey merged 9 commits into
mainfrom
port/sp4-a-test-isolation
Oct 16, 2026
Merged

donky-ey merged 9 commits into
mainfrom
port/sp4-a-test-isolation

Conversation

@donky-ey

@donky-ey donky-ey commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

First PR of the SP4 upstream port (batch A of A–H). It carries the SP4 triage spec and plan, plus the test-isolation batch that every later batch's tests rely on.

  • Spec and plan. docs/superpowers/specs/2026-10-16-sp4-upstream-port-design.md triages all 191 commits in main..upstream/main (v2.5.6) and records the confirmed scope. docs/superpowers/plans/2026-10-16-sp4-upstream-port.md lays out batches A–H, one PR each, in the order A → D → G → B → C → E → F → H.

  • Isolate every app-support store from test runs (upstream fix: stop the test suite from reading and rewriting the user's usage cache chattymin/PokeTokenBar#359 + class sweep). swift test uses the same storage name as the installed app, so code that reached a store's default path read or wrote the real ~/Library/Application Support/PokeDexBar/. Upstream fix: stop the test suite from reading and rewriting the user's usage cache chattymin/PokeTokenBar#359 fixed this for usage-cache.json with a per-store gate. A sweep found the same class in more places:

    Store Before this PR, in a non-bundled run
    usage-cache.json read and rewritten (the default providers use LocalUsageCache.shared)
    sprite cache written: FormTests put fake GIFs into it
    player-state.json, base-index.json, weather.json safe only because every test happened to inject a path
    session-key.json already redirected on its own (the precedent)

    Instead of gating each store, the fix isolates at the root. For non-bundled runs, AppEnv.supportDirectory() returns a temporary folder named with a UUID, fresh for each run, and every Application Support path now goes through it. AppEnv.userSupportDirectory names the real location for test comparisons without creating it. The session-key store now shares the root.

  • Guards. Each one catches a different change; all were mutation-checked.

    • AppStorageIsolationTests pins the root branch and each store's default path. Reverting the root branch fails both, plus the session-key location test (11 failures).
    • Its source scan (testNoAppSupportPathIsComposedOutsideAppEnv) forbids applicationSupportDirectory and userSupportDirectory outside AppEnv.swift. It does not react to a reverted root branch. It fails when a store composes the path itself (checked by restoring the old LocalUsageCache.defaultFileURL) or uses userSupportDirectory.
    • The upstream regression test is ported as testDefaultPathCacheNeverRewritesTheUserFile.
  • Screenshot generator. It clones the installed app's sprite cache read-only into the isolated cache, so its renders still have sprites.

  • test: pin the locale when asserting grouped numbers chattymin/PokeTokenBar#166. TokenFormatter.grouped takes a locale (default .current, so production is unchanged), and the test pins en_US and es_ES.

  • fix: route the Antigravity session notice through the localization table chattymin/PokeTokenBar#222 (test only). LocalizedUILiteralTests fails on Hangul string literals in UI/ and PokeDexBarApp.swift. There are 0 today, and a planted literal is caught.

  • Two test fixes found by the gate.

    • testEveryStoredPathDerivesFromTheNamespace now checks the bundled path.
    • testPayloadIsNotReadableInTheFile stopped searching random AES-GCM ciphertext for "42". That matched by chance in about 1.5% of runs.
  • CLAUDE.md. The test-storage rule now covers all app storage, not only credentials.

Upstream commits: chattymin#359 (d6d24a0), chattymin#166 (f9ae41d), chattymin#222 (a6ed409, test only; the Antigravity notice it shipped with does not exist here).

Trade-off vs. upstream chattymin#359

Upstream gates each store (persistsToUserLocation, persistsToDisk, cachedBlobCount == 1). With that design, a default-path store under swift test reads and writes nothing, so fixtures can never mix with user-derived data. This PR redirects the root folder instead. It closes the class for every current and future store, but within one test process the default-path stores share the same temporary files. For example, an earlier UsageStore() refresh can fill the shared temporary usage-cache.json from the developer's real ~/.claude logs. The real user files are never read or written, but upstream's "no user-derived data next to fixtures" guarantee does not hold. Tests that assert on cache contents inject a path (as this suite already does). Accepted by design; CLAUDE.md records it.

Review changes (61666c7)

  • @discardableResult is back on writeFile. It had drifted onto a test function and caused 41 warnings.
  • The scan now also forbids userSupportDirectory outside AppEnv.swift. Its comment stripping walks characters through a shared SourceScan helper, so a // inside a string no longer cuts the line.
  • The isolated folder name uses a UUID instead of the PID, which can be reused and inherited.
  • I corrected the guard descriptions above and in CLAUDE.md. The earlier text said all three tests fail when the root branch is reverted; the scan does not.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation
  • Other:

UI changes

None visible. UI/SpriteLoader.swift only changes how its cache path is derived. I ran the screenshot generator (PTB_SCREENSHOTS=1) on this branch: every sprite-heavy static PNG came out byte-identical to the committed assets. Only the animated GIFs and two time-dependent PNGs differed, and I discarded those regenerated files.

Verification

  • ./scripts/test-gate.sh (after the review changes):

    • Executed 1992 tests, with 11 tests skipped and 0 failures (0 unexpected)
    • ✓ 게이트 통과 — 로직 코어 라인 커버리지 94.18% >= 75%
  • Mutation checks:

    Change Fails
    Root branch reverted root-branch (2) + per-store (6) + session-key (3)
    Old defaultFileURL composition restored per-store + scan
    A store using userSupportDirectory per-store + scan
    Hangul literal planted after a "https://…" string LocalizedUILiteralTests
  • Real-folder check: the installed app was running during the gate and rewrites usage-cache.json about once a minute on its own. I observed that by watching the file for 7 minutes with no tests running. So "unchanged mtime across the suite" cannot be observed on this machine. The isolation is proven by the path tests instead.

Hotspot files

  • Core/UsageStore.swift: 2 lines (the parity snapshot now uses AppEnv.supportDirectory(); it was already bundled-only).
  • UI/PopoverView.swift, Core/Localization.swift, UI/SettingsView.swift: not touched.

Checklist

  • swift build and swift test pass locally
  • PR title and description are written in English
  • UI changes are described above (before/after — images optional)
  • No copyrighted assets, secrets, or private tooling references are committed (see CONTRIBUTING)
  • Tests were added or updated for this change

Classify all 191 non-merge commits in main..upstream/main (v2.5.6, 77f789d):
56 already in the fork, 87 not applicable, 26 port candidates grouped into
batches A-H, 22 needing a user decision (new providers, small UX features,
save backups). Also checks open upstream issues chattymin#337, chattymin#443 and chattymin#439 against
the fork's code; all three apply.
The user confirmed the triage recommendations: new providers, the pet size
ceiling, variable menu-bar width, the Low Power GIF policy and save backups
stay out; the right-click menu (chattymin#373) joins batch E; chattymin#337 and chattymin#443 are in,
with chattymin#443 skipping claude-code mirror rows unconditionally. chattymin#216 is deferred.
Eight batches (A-H), one PR each, executed A -> D -> G -> B -> C -> E -> F -> H.
…in#359)

`swift test` uses the same storage name as the installed app, so any code
that reached a store's default path read or wrote the user's real
Application Support folder. Upstream chattymin#359 measured usage-cache.json being
rewritten by one suite run. A sweep found the same class in more places:
the sprite cache (a test wrote fake GIFs into it), and the save,
base-index.json and weather.json, which were safe only because every test
happened to inject a path.

Instead of gating each store, isolate at the root: for non-bundled runs
AppEnv.supportDirectory() returns a per-process temporary folder, and
every Application Support path now goes through it. The session-key store
already did this on its own and now shares the root. AppEnv.userSupportDirectory
names the real location for comparisons without creating it.

Guards: AppStorageIsolationTests pins the root branch, each store's default
path, and a source scan that forbids composing Application Support paths
outside AppEnv. All three fail when the isolation branch is reverted. The
upstream regression test is ported as testDefaultPathCacheNeverRewritesTheUserFile.
The screenshot generator clones the installed app's sprite cache read-only
into the isolated cache so its renders still have sprites.

CLAUDE.md: the test-storage rule now covers all app storage, not only
credentials.
…in#166)

TokenFormatter.grouped follows the system region, as macOS does, but the
test hardcoded comma grouping, so it failed on machines whose region groups
digits differently. grouped now takes a locale defaulting to .current
(production unchanged); the test pins en_US and adds an es_ES case that
fails if the argument is ignored.
…ymin#222)

Interpolation tests only see strings that already go through L, so a
literal that never reaches the table is invisible to them. Scan the UI
sources (plus the app entry file, which holds UI copy in this fork) and
fail on any Hangul string literal; Korean comments stay allowed. Core and
Player are left out because their per-language data tables hold Hangul
legitimately. Only the test is ported; the Antigravity notice it shipped
with does not exist here.
…ated

testEveryStoredPathDerivesFromTheNamespace asserted that the support folder
ends in the storage name, which held only because swift test used the real
folder. Check the bundled path with the pure supportDirectoryURL(isBundledApp:)
and keep the save-under-the-root check for the running process.
testPayloadIsNotReadableInTheFile searched the envelope for "42". The seal
uses a fresh AES-GCM nonce each time, so those two characters show up in
the base64 by chance in roughly 1.5% of runs, which failed the gate
intermittently. Assert that the plaintext payload bytes are absent instead.
…on (review)

- Put @discardableResult back on writeFile. The new tests had been inserted
  between the attribute and the helper, which added 41 unused-result warnings.
- The source scan now also forbids AppEnv.userSupportDirectory outside
  AppEnv.swift. A future store could otherwise reach the real folder through
  the new accessor and pass every guard.
- Comment stripping in the scan walks characters (shared SourceScan, also
  used by LocalizedUILiteralTests), so a "//" inside a string no longer hides
  the rest of the line.
- The isolated folder is named with a UUID instead of the PID. PIDs are
  reused, so a later run could inherit an earlier run's cache.
- Correct the claim in 6b52c9e and CLAUDE.md that all three isolation tests
  fail when the root branch is reverted. Mutation checks show: reverting the
  root fails the root-branch and per-store tests (plus the session-key test);
  the source scan does not react to that, and instead fails when a store
  composes the path itself (the old LocalUsageCache.defaultFileURL, restored)
  or uses userSupportDirectory.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants