Skip to content

Policy: TK-M003 takes about 100 ms on 64 KiB in the fallback engine (a slow-runner flake) #699

Open

Description

@LoopGlitch26

tests/test_policy2.py::Speed::test_every_pack_pattern_searches_64_kib_within_budget failed on CI's py3.13 macOS runner for #642, which doesn't change TK-M003:

TK-M003 pattern 104 ms on 'cc":; "to":[""," "to":["'

TK-M003 (tracekit/policy2/packs/server-comms.yaml:61: ten or more recipients, or a whole-company address, in to, cc, bcc or recipients) takes about 100 ms on 64 KiB of that repeated text in the regex fallback on a slow runner. The test (tests/test_policy2.py:699) holds TK-M003 to its own budget, 60 ms scaled by how much slower the runner is (RULE_BUDGET_S, :630; machine_scale(), :920), and 100 ms is right at it. So the test fails now and then. #482 (for #481) already made the pattern read a recipient list forward instead of backing off over it.

What's wanted: a TK-M003 pattern that decides the same calls but stays well under the budget on 64 KiB, for example by bounding the repetition that scans recipient lists. Today the worst filler takes about 37 ms on an M-series Mac (the "~12 ms" in the comment at tests/test_policy2.py:628 is out of date).

Checks:

  • RULE_BUDGET_S["TK-M003"] lowered from 0.06 to 0.02 in the same change, and Speed still passes on both engines (update the comment at :628 with the new time);
  • Same verdicts on E17's corpus and tests/data/policy (verdict diff against main: nothing loosened);
  • E17 passes, and eval/e21_robustness.py passes without --rebaseline.

Comment here to claim it. Commits need a DCO sign-off (git commit -s).

Activity

  1. dbdefault commented on Nov 10, 2026

    @dbdefault

    I'd like to take this one. Plan: bound the recipient-list repetition in TK-M003 so it decides the same calls, lower RULE_BUDGET_S[TK-M003] to 0.02 with the updated timing comment, and check verdicts against E17's corpus and tests/data/policy plus E17/E21 as listed. Commits will carry a DCO sign-off.

  2. LoopGlitch26 commented on Nov 10, 2026

    @LoopGlitch26
    MemberAuthor

    Assigned to you, thanks; the plan matches the issue. Since this is a policy rule, the review will compare every decision under main's server, strict and dev packs against yours on both regex engines, so please include in the PR the commands you used for the verdict diff and the before and after timings on the worst filler. A version bump of server-comms.yaml goes with the pattern change.

  3. dbdefault commented on Nov 10, 2026

    @dbdefault

    I'm releasing this one so it doesn't stay blocked on me. I couldn't find a change that keeps every verdict and meets the 0.02 budget in the fallback engine. Notes in case they help whoever picks it up:

    • The fallback is the third-party regex module. On 65,532 chars of the reported filler (Windows, local): full pattern 61.5 ms, count branch 5.6 ms, whole-company branch 55.5 ms. RE2 is ~0.1 ms for all three.
    • Cost grows about linearly (15.4 / 32.2 / 61.5 ms at 16 / 32 / 64 KiB). The filler produces ~5,461 "to": start positions, and the repeated whole-company/list/string attempts at each start dominate. It is not one quadratic rescan of a long list.
    • Factoring the alternatives and dropping captures got to ~39 ms. Atomic groups and possessive quantifiers would help, but they fall outside the shared RE2 subset.
    • Capping the list scan isn't safe. Both engines currently return ask/TK-M003 for a whole-company address that follows 10,000 numeric list elements, so a cap would loosen that verdict.
  4. LoopGlitch26 commented on Nov 10, 2026

    @LoopGlitch26
    MemberAuthor

    Thanks for the measurements and for saying so plainly; they save the next person a lot of time. The 10,000-element case is the important one: it rules out capping the list scan. I've unassigned you, and the issue is open again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions