Skip to content

Trials P-17 (fail) and P-22 (pass): a Run is identified from the run records - #162

Merged
don-petry merged 2 commits into
mainfrom
trials/p-17-p-22
Oct 18, 2026
Merged

don-petry merged 2 commits into
mainfrom
trials/p-17-p-22

Conversation

@don-petry

@don-petry don-petry commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes #10.

Trials P-17 and P-22: can a Run be identified from the run records? (design §6, batch 1; I-21, I-38)

Trial Pass condition, written first Verdict
P-17 A run's name is set when it is created and is visible for a run whose job never started; recent runs of one workflow can be listed and matched by name, at a recorded cost Fail on the first half, pass on the second
P-22 A started Run is in the list within one pass interval. The bound was taken as 60 s, the shorter end of the relay's "every minute or two", since §7.3 gives no single interval. Measured over at least 20 starts; "no" if any exceeds it Pass

P-17

  • First appearance: a new run is in the list within about 1.6 s. But the first read that holds it shows the workflow's name, not the run-name, in 10 of 10 starts whose job never started and in 13 of 24 hosted starts. The run-name appears about a second later, by 3.7 s at most.
  • A run whose job never starts stays listed, queued and named, until it is cancelled (5 minutes in the trial).
  • Listing by name: one request lists a workflow's recent runs, and the run is matched by name. That costs 1 request, and 1 point of the 5,000-an-hour core limit.
  • The failure is filed against the design as Design: a Run is identified before its run-name is listed (P-17 failed) #161, "Design: a Run is identified before its run-name is listed (P-17 failed)", before any code depends on it.

P-22: across 24 starts, every run was in the list at the first read, 2 s after the dispatch. Time to that read's response: min 2.28 s, median 2.34 s, max 2.80 s. These are upper bounds.

Also seen, recorded in #161:

  • The run-list endpoint answers with cache-control: max-age=60.
  • The dispatch reply carries Deprecation and Sunset headers.
  • Reading a run's jobs counts against a separate rate-limit window.

What changes

  • trials/P-17.json and trials/P-22.json. They hold full exchanges for a sample of starts, plus a timing table copied from the redacted captures. Every request in full would have run to megabytes.
  • Grounding in sim/grounding.py:
    • a_queued_run_is_listed names P-17.
    • a_new_run_is_listed_late names P-22.
    • a_superseded_run_fails stays unverified: a cancelled run that never started concludes cancelled, not failure.
  • One test fix the grounding needed: test_verify_prints_every_rule_no_fixture_grounds assumed no rule was grounded yet. It is in a commit of its own.

Redaction removed the authorization header on every request, listed in removed, and 13 response headers a fixture may not keep. Nothing shaped like a credential was found.

Step 8 undid:

  • the 11 queued runs, cancelled;
  • the 35 trial runs, deleted;
  • the trial workflow, disabled.

There were no branches or PRs. Staying in the canary: the workflow commit on main and one CodeQL run. As in P-3, the environment main-only and its real secret were never named, read or touched.

Verified: python3 -m tools.check_trials passes, and make verify passes on a git archive export. The mutation run is CI's.

🤖 Generated with Claude Code

View guided diff Turn on auto-fix

don-petry and others added 2 commits October 17, 2026 20:38
…orkflow's name first

P-17 fails. A run started by workflow_dispatch is in its workflow's list
about 1.6 s after the dispatch is sent, and a run whose job never starts
stays listed, queued, until cancelled. But the first read that holds the
run names it after the workflow, not its run-name, in 10 of 10 never-
starting starts and 13 of 24 hosted ones. The run-name follows about a
second later. One request lists a workflow's recent runs, and they match
by name, at one point of the core rate limit.

P-22 passes. Over 24 starts, the run was in the list at the first read,
2 s after each dispatch: 2.28 s, 2.34 s and 2.80 s to the response, at
minimum, median and maximum, against a bound of 60 s.

Grounds a_queued_run_is_listed in P-17 and a_new_run_is_listed_late in
P-22. a_superseded_run_fails stays unverified: a never-started run that
is cancelled concludes cancelled, and a Run that stops itself was not
tried.

Part of #10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d ones

The test counted every rule of the fake as unverified, which held only
while no fixture grounded any. The first grounding, P-17's and P-22's,
broke it. It now expects exactly the rules with no fixture, and checks
that a grounded rule is not printed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner October 18, 2026 01:49
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-18T01:49:24.080809Z 0fc9a8d PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 13 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: combsmith/software-factory/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c6e25b77-23d3-4cad-b23d-cf4d6300a2e1
📥 Commits

Reviewing files that changed from the base of the PR and between ab22a1e and 0fc9a8d.

📒 Files selected for processing: 4
  • sim/grounding.py
  • tests/test_tools.py
  • trials/P-17.json
  • trials/P-22.json
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@don-petry don-petry left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial Code Review against config/rubric-pr-review.json

  • PR-1 (Requirements, Major): PASS. Implements Trial P-17 (fail) and P-22 (pass) for Run identification under issue #10. Correctly captures the first read workflow-name latency in P-17 (tracked in #161) and records P-22 list latency within bounds.
  • PR-2 (Security, Blocking): PASS. Authorization and sensitive response headers properly redacted in both fixtures.
  • PR-3 (Functional, Major): PASS. Grounds a_queued_run_is_listed and a_new_run_is_listed_late in sim/grounding.py; updated test_verify_prints_every_rule_no_fixture_grounds to distinguish ungrounded vs grounded rules. Verified via tools.check_trials and tests/test_tools.py::Trials.
  • PR-4 (Architecture, Blocking): PASS. No changes under kernel/.
  • PR-5 (Requirements, Major): PASS. No changes to decisions or AGENTS.md.

Verdict: PASS (Zero Critical, High, or Medium findings).

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trials P-17 and P-22: a Run can be identified from the run records

1 participant