Skip to content

Triage of the failing test suite: 74 tests, grouped by cause #1278

Closed

Description

@gHashTag

Triage of the failing test suite: 74 tests, grouped by cause

Triaged by cause rather than by file. Three product defects were fixed in fa904a3bb; the rest are catalogued here.

Three numbers that are not the same number

The summary line counts assertion failures, not tests. "104 failures" was 74 failing tests. Earlier reports quoted 122, 118, 105 and 99 interchangeably - all assertion counts. Use grep -c "^Test Case .* failed" on unique lines.

Remaining, and what every one of them has in common

Every class sampled fails in isolation as well as in the full run. Two classes fail MORE alone than together, which is order dependence in the direction nobody looks for.

Class full run alone
LogsTabViewTests 12 13
LocalAuthProviderTests 9 13
ModelHealthServiceTests 8 9
ChatFailureTests 5 6
ModelConfigurationStoreCrossProviderTests 4 6
WarmupVolatilityTrackerTests 3 3
MemoryStoreFTSTests 3 3
StreamingContextWatchdogTests 2 2
ConversationEncryptionTests 2 2

What a bee can do without a Swift compiler

The worker container cannot run swift test. It can make the counting honest: a script that takes an xcodebuild/swift test log and reports failing TESTS, not assertions, per class, with the isolation column filled from a second log. Then the next triage starts from a number that means one thing.

User Scenarios & Testing

User Story 1 - One log, one number that means "failing tests" (P1)

Acceptance Scenarios:

  1. Given a swift test log with 104 assertion-failure lines across 74 distinct Test Case ... failed lines,
    When the script runs,
    Then it prints 74, not 104, and one row per class with its count.
  2. Given two logs, a full run and a per-class run,
    When the script runs with both,
    Then it prints the table above with the alone column filled and flags every class whose alone count exceeds its full count.

Requirements

  • FR-001: The script MUST count unique Test Case '...' failed lines, never XCTAssert lines.
  • FR-002: The script MUST accept one or two log paths and MUST NOT run any build or test itself.
  • FR-003: Output MUST be a markdown table with columns class, full, alone, worse alone.
  • FR-004: A fixture log MUST be committed so the script is testable without Xcode.

Success Criteria

  • trios/tools/xctest-triage.mjs exists and runs under node with no dependencies outside the Node standard library (the worker image carries node and bun; it carries no python3).
  • trios/tests/fixtures/xctest-triage-sample.log exists and contains at least 74 distinct Test Case failure lines and at least 100 assertion lines.
  • Running node trios/tools/xctest-triage.mjs trios/tests/fixtures/xctest-triage-sample.log prints 74 as the failing-test total; the output is quoted in the bee's closing report.
  • The script exports a function named failingTestsByClass; that identifier appears nowhere in the tree today.

Boundary

trios/tools/xctest-triage.mjs
trios/tests/fixtures/xctest-triage-sample.log

Activity

  1. gHashTag commented on Sep 4, 2026

    @gHashTag
    CollaboratorAuthor

    Closed after verifying the work exists.\n\nThe Queen accepted this dispatch. The result is commit on branch - feat(tools): add xctest-triage to count failing tests, not assertions - now pushed to .\n\nIt stayed open because the deployed supervisor cannot write to GitHub: makes exactly two calls, both GET, and the container had no git credential configured. 100 branches of accepted work were sitting on the Railway volume unpushed, and 76 finished issues like this one stayed open and jammed the candidate pool - the round skipped them as every tick. The branches are pushed now; the missing write path is the remaining defect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions