Skip to content

Automated-test documents no longer run as written #411

Description

@DutchJaFO

Description

Five automated-test documents no longer run as written, found while running them for #370's T2 pass on 2026-09-18 and #409's on 2026-09-19:

  • A file left awaiting review raises an alert, and resolving it retires the alert (import-and-staged-actions/20-pending-review-alert.md) — step 6 expects three alerts, one of them obsolete, after two reseeds. Since Reseed should only import the designated files, not delete data first #372 a reseed no longer truncates a batch, and since A Source-level Modify conflict stages a new Pending action on every reseed instead of staying stable #376 an already-reported conflict is not re-staged, so no alert can become obsolete: the run shows two alerts, one active and one resolved. Step 8's "No longer applicable" half is unreachable for the same reason. Step 7 also destroys qt-review-20 before steps 8 and 9 drive it in the browser, so the document cannot run in its written order.
  • An already-reported conflict does not stage a duplicate on every reseed (import-and-staged-actions/25-an-already-reported-conflict-does-not-accumulate.md) — steps 4 and 5 print empty counts instead of 1. Its Unresolved function returns .Count on a filtered result that is a single object, which PowerShell 5.1 unrolls, so .Count is empty — the pitfall the pending-review alert document itself warns about. The behaviour under test is correct: step 6 reads AlreadyReported=2 and Modify=1.
  • Bulk-deciding a staged batch via file export and re-import, in both wire formats (import-and-staged-actions/13-bulk-decide-via-file-export-import.md) — step 2 imports the curated file under review and expects 202. Since An import that re-states identical content reports it as modified #373 an already-stored file stages nothing, so it answers 200 and the document never reaches the bulk-decide round trip it exists for — the reason The staged review → decide → apply workflow moved to the conflict fixture.
  • The changelog is served from its own on-disk database, not the JSON fallback (notifications-and-changelog/07-changelog-served-from-its-own-database.md) — step 5 reads the log as soon as health answers after a restart, but the changelog import runs after that: its refreshed line appeared 716 ms later. The step then reports no import.
  • Reset wipes the entire database and does not reseed (database-lifecycle/03-reset-is-a-full-wipe.md) — the cleanup removes smoke156-before.db but not the -wal/-shm files DbInspector creates beside it when step 7 reads it.

Running them for this issue on 2026-09-19 found four more ways a run reports something the test did not cause:

  • The pending-review alert document writes into its bind folder without clearing it. A folder an earlier run left behind still holds that run's database, so step 6 counted four alerts.
  • The already-reported conflict document reseeds straight after docker restart, before the application answers; the reseed fails with the connection closed.
  • Every browser-driven document logs a CryptographicException / AntiforgeryValidationException pair it did not cause. Each test container starts with its own DataProtection key ring, and the browser pane keeps a cookie from an earlier container — it does not separate cookies by port, nor by *.localhost host name. Measured: 2 to 7 such lines on a first visit, and 0 when both containers read one shared key ring.
  • Exception counts included the application's own shutdown. Stopping the application logs a SocketException per listening port, and an OperationCanceledException per connected page; a log read after a stop counts them with the test's own.

Reproduction steps

Run each document exactly as written against a current quotinator:local.

Expected behaviour

  1. The pending-review alert document's step 6 asserts what a reseed now produces, and step 8 reaches both dismiss reasons through a path that still exists; if none does, the obsolete half moves to a test that constructs the state. Steps run in their written order.
  2. The already-reported conflict document's steps 4 and 5 print 1 -> 1 -> 1.
  3. The bulk-decide document stages a batch that has something to decide, and runs its round trip.
  4. The changelog database document's step 5 reads the import line once it has been written.
  5. The reset document leaves nothing behind in .claude/temp.
  6. The pending-review alert document runs against a fresh database, whatever an earlier run left behind.
  7. The already-reported conflict document reseeds only once the restarted application answers.
  8. A test logs the antiforgery pair only when it sets out to provoke it. Test containers share one key ring by default; a test can opt out, and a read-only data directory never shares it. A test that provokes the pair ends by restoring a cookie the shared ring can read, and proves the next visit logs nothing.
  9. A container's log is read for exceptions before the application stops, except in a test about shutdown.

Actual behaviour

  • Pending-review alert, step 6: cc95b26e isDismissed=False, 9818586e isDismissed=True reason=resolved, active alerts = 1 — two alerts, none obsolete.
  • Already-reported conflict, steps 4–5: after cold start: and -> ->.
  • Bulk decide, step 2: < 200 OK, Expected 202, got 200.
  • Changelog database, step 5: after the restart only [Changelog - Init] schema is up to date (v1); the [Changelog - Import] refreshed line follows later.
  • Reset: smoke156-before.db-shm and smoke156-before.db-wal remain in .claude/temp.
  • Pending-review alert, over a folder an earlier run left: step 6 lists four alerts, two of them resolved, created before the container started.
  • Already-reported conflict, step 3: The underlying connection was closed: The connection was closed unexpectedly.
  • Notifications list, dismiss, render, and drive their action: CryptographicException and AntiforgeryValidationException before its first stop, and 7 more lines logged by the stop itself.

Failing tests

Test class Test method Status before fix
A file left awaiting review raises an alert, and resolving it retires the alert Steps 6 and 8 as written ❌
An already-reported conflict does not stage a duplicate on every reseed Steps 4 and 5 as written ❌
Bulk-deciding a staged batch via file export and re-import, in both wire formats Step 2 as written ❌
The changelog is served from its own on-disk database, not the JSON fallback Step 5 as written ❌
Reset wipes the entire database and does not reseed Cleanup as written ❌
A file left awaiting review raises an alert, and resolving it retires the alert Step 1, over a folder an earlier run left ❌
An already-reported conflict does not stage a duplicate on every reseed Step 3 as written ❌
Notifications list, dismiss, render, and drive their action No [Runtime - Exception] line before its first stop ❌
A file left awaiting review raises an alert, and resolving it retires the alert No [Runtime - Exception] line before step 9 ❌
A new document provoking the antiforgery pair The pair is logged once; after the restore, the next visit logs nothing ❌

Definition of done

  • Failing test(s) listed above are red before the fix is written
  • Fix implemented
  • All listed tests pass (green)
  • No regression in related tests
  • Findings summarised in a closing comment

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions