Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/canary-rollout.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ on:
- persona-mention
- pr-auto-review
- pr-review-mention
- solution-architect
default: dev-lead
override:
description: "promote: advance even if the gate is not PROMOTE (records a warning)"
Expand Down
2 changes: 1 addition & 1 deletion node_modules/.package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

65 changes: 65 additions & 0 deletions standards/canary-rings.json
Original file line number Diff line number Diff line change
Expand Up @@ -1078,6 +1078,71 @@
}
}
}
},
"solution-architect": {
Comment thread
don-petry marked this conversation as resolved.
"_note": "Onboarded per persona-standards.md section 6 step 2 (petry-projects/.github#1078). This persona has NO dedicated reusable: it rides the SHARED, convention-driven persona runtime -- persona-runner-reusable.yml on petry-projects/.github-private, dispatched by the persona-mention router -- which runs any persona that supplies a manifest plus prompts/<role>/advisory.md. The reusable registered here is therefore that shared runner (the reusable that actually executes this persona), and rings[]/gate mirror agents.persona-mention so ring membership and soak knobs stay consistent with the runtime that serves it. CONSEQUENCE 1 (autocut): autocut will NOT fire on this persona's own prompt-only changes -- agent_ref_paths is overridden below to [\"scripts/\",\"personas/\"] (explicitly excluding prompts/) so the global fleet default (which includes prompts/) does not apply here; a prompt-only change therefore never moves either the runner's blob OR the agent-level watched paths (petry-projects/.github-private#1592 -- Script-only changes never reach the fleet). A ring label is therefore NOT evidence of promotion; scripts/persona_reach_check.sh in .github-private is the compensating control that separates routes-to from promoted and fails on the skew state. CONSEQUENCE 2 (fleet precedent): eight other personas are drafted behind this same shared runtime; if each registers against this reusable, several agents end up watching one blob and one change cuts several versions -- decided deliberately here, not by accumulation. NOTE (gate sampling): run_workflow='Persona Runtime' is shared across all personas on this runner; the gate therefore samples all persona runs against the shared runner blob -- this is intentional (the gate tracks runner health, not per-persona prompt health) and is documented here so the deliberate design is not mistaken for an oversight.",
"host": "petry-projects/.github-private",
"reusable": ".github/workflows/persona-runner-reusable.yml",
"run_workflow": "Persona Runtime",
Comment on lines +1085 to +1086

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Persona Runtime is shared by multiple personas, but the gate filters only by repository and workflow name, so unrelated persona runs contaminate solution-architect sampling and failure counts. [api mismatch]

Assessment: 🟠 Major · 🔁 Occurrence: Sometimes

Use CodeAnt Skill Fix in Cursor Fix in VSCode Claude

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** standards/canary-rings.json
**Line:** 1085:1086
**Comment:**
	*Api Mismatch: `Persona Runtime` is shared by multiple personas, but the gate filters only by repository and workflow name, so unrelated persona runs contaminate solution-architect sampling and failure counts.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged and documented explicitly. The shared run_workflow: "Persona Runtime" sampling all personas' runs against the shared runner blob is intentional per CONSEQUENCE 2 — the gate tracks runner health, not per-persona prompt health. The _note now contains an explicit NOTE section stating this is a deliberate design choice (not an oversight), so it is clear to future reviewers.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Customized review instruction saved!

Instruction:

Do not flag shared workflow sampling across personas when the canary gate is intentionally tracking shared runner health rather than per-persona prompt health.

Applied to:

  • standards/canary-rings.json

💡 To manage or update this instruction, visit: CodeAnt AI Settings

"agent_ref_paths": ["scripts/", "personas/"],
"rings": [
{
"channel": "next",
"order": 0,
"members": [
"$host"
]
},
{
"channel": "ring0",
"order": 1,
"members": [
"$org_infra"
]
},
Comment thread
don-petry marked this conversation as resolved.
{
"channel": "ring1",
"order": 2,
"members": [
"petry-projects/TalkTerm",
"petry-projects/bmad-bgreat-suite"
]
},
{
"channel": "stable",
"order": 3,
"members": [
"*"
]
}
],
"gate": {
"_standard": "petry-projects/.github#548 — graduated dwell/sample gate over a per-candidate cumulative window. These are registry-configurable per-transition knobs; the values below are the #548 defaults.",
"baseline_window_days": 14,
"baseline_spike_cap_multiple": 3,
"benign_failure_classes": [],
"control": {
"allow_pre_existing": false
},
"_benign_note": "benign_failure_classes (#1025 P2): per-reusable allowlist matched against a failed run's workflow name (`workflow` ERE) + failed-step signature (`step` ERE, matched via `gh run view`). Matches are excluded from cum_fail — but ONLY when the candidate's reusable is byte-identical to the prior channel (differs=0), so the allowlist can never mask a candidate-introduced regression. Exception (#668): a class marked `version_independent: true` fails before/independent of the candidate's own code (e.g. a Dependabot-context secrets failure at startup) and can NEVER be candidate-caused, so it is excluded even when differs=1 — reserve the flag for classes where that is provable from the failure mechanism, not merely likely. `control.allow_pre_existing`, or `promote --allow-pre-existing`, advances a BLOCKED+PRE_EXISTING frontier (never REGRESSION).",
"transitions": {
"next->ring0": {
"dwell_hours": 4,
"sample_fraction_permille": 250,
"sample_clamp_min": 3,
"sample_clamp_max": 15,
"waive_sample_if_no_caller": true
},
"ring0->ring1": {
"dwell_hours": 8,
"waive_sample": true
},
"ring1->stable": {
"dwell_hours": 12,
"sample_min": 1
}
}
}
}
},
"unmanaged": {}
Expand Down
46 changes: 46 additions & 0 deletions tests/canary_rollout.bats
Original file line number Diff line number Diff line change
Expand Up @@ -862,6 +862,52 @@ GHEOF
[ "$status" -eq 0 ]
}

@test "canary-rings.json: solution-architect onboarded on the shared persona runtime (no dedicated reusable, #1078)" {
# Registered against the SHARED, convention-driven persona runtime — persona-runner-reusable.yml
# in .github-private (the reusable that actually executes this persona), NOT a per-persona blob.
run jq -e '.agents["solution-architect"].host == "petry-projects/.github-private"' "$RINGS"
[ "$status" -eq 0 ]
run jq -e '.agents["solution-architect"].reusable == ".github/workflows/persona-runner-reusable.yml"' "$RINGS"
[ "$status" -eq 0 ]
# run_workflow = the persona-runner.yml caller display name the gate samples on the ring tiers.
run jq -e '.agents["solution-architect"].run_workflow == "Persona Runtime"' "$RINGS"
Comment on lines +872 to +873

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: The test approves a shared workflow name without checking persona filtering, so unrelated Persona Runtime runs can contaminate solution-architect health decisions while all assertions pass. [api mismatch]

Assessment: 🟠 Major · 🔁 Occurrence: Sometimes

Use CodeAnt Skill Fix in Cursor Fix in VSCode Claude

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** tests/canary_rollout.bats
**Line:** 872:873
**Comment:**
	*Api Mismatch: The test approves a shared workflow name without checking persona filtering, so unrelated Persona Runtime runs can contaminate solution-architect health decisions while all assertions pass.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test correctly asserts run_workflow == "Persona Runtime" because that is the actual value — the design is intentional (gate tracks shared runner health across all personas). The _note has been updated in standards/canary-rings.json with an explicit NOTE section documenting that the shared workflow name is by design, so this is no longer an undocumented gap.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Customized review instruction saved!

Instruction:

Do not require persona-specific filtering when canary tests intentionally validate the shared Persona Runtime health across all personas.

Applied to:

  • tests/canary_rollout.bats

💡 To manage or update this instruction, visit: CodeAnt AI Settings

[ "$status" -eq 0 ]
# standard ring model
run bash -c "jq -r '.agents[\"solution-architect\"].rings | sort_by(.order) | map(.channel) | join(\",\")' '$RINGS'"
[ "$output" = "next,ring0,ring1,stable" ]
# AC #3: next = the dogfood ring ($host = .github-private), matching agents.persona-mention
run jq -e '.agents["solution-architect"].rings[] | select(.channel=="next") | .members | index("$host")' "$RINGS"
[ "$status" -eq 0 ]
# ring0 = org_infra repos ($org_infra), consistent with dev-lead and ci-failure-analyst
run jq -e '.agents["solution-architect"].rings[] | select(.channel=="ring0") | .members | index("$org_infra")' "$RINGS"
[ "$status" -eq 0 ]
# ring1 = the real consumers that gate ring1->stable (consistent with persona-mention)
run jq -e '.agents["solution-architect"].rings[] | select(.channel=="ring1") | (.members|index("petry-projects/TalkTerm")) and (.members|index("petry-projects/bmad-bgreat-suite"))' "$RINGS"
[ "$status" -eq 0 ]
# gate mirrors agents.persona-mention: #548 defaults, standard transition knobs, no opt-in blocks
run jq -e '.agents["solution-architect"].gate.baseline_window_days == 14 and .agents["solution-architect"].gate.baseline_spike_cap_multiple == 3' "$RINGS"
[ "$status" -eq 0 ]
run jq -e '.agents["solution-architect"].gate.transitions["next->ring0"].waive_sample_if_no_caller == true and .agents["solution-architect"].gate.transitions["ring0->ring1"].waive_sample == true and .agents["solution-architect"].gate.transitions["ring1->stable"].sample_min == 1' "$RINGS"
[ "$status" -eq 0 ]
run jq -e '.agents["solution-architect"].gate.transitions["ring1->stable"].dwell_hours == 12' "$RINGS"
[ "$status" -eq 0 ]
# no opt-in gate machinery (like persona-mention): no correctness, no suspect classes, empty benign list,
# no synthetic-canary fields
run jq -e '.agents["solution-architect"].gate | (has("correctness")|not) and (has("suspect_failure_classes")|not) and ((.benign_failure_classes // []) | length == 0)' "$RINGS"
[ "$status" -eq 0 ]
run jq -e '.agents["solution-architect"] | (has("next_tier_health_signal")|not) and (has("soak_start_ring")|not)' "$RINGS"
[ "$status" -eq 0 ]
# AC #2: inline _note documents (a) no dedicated reusable / shared runtime, and (b) autocut does
# not fire on prompt-only changes — naming .github-private#1592 and the persona_reach_check.sh control.
run jq -e '.agents["solution-architect"]._note | test("1592") and test("persona_reach_check.sh") and test("shared"; "i")' "$RINGS"
Comment on lines +900 to +902

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: The test only checks that the note claims prompt changes do not autocut; it never exercises autocut, so it passes while prompts/ changes still create candidates. [possible bug]

Assessment: 🟠 Major · 🔁 Occurrence: Sometimes

Use CodeAnt Skill Fix in Cursor Fix in VSCode Claude

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** tests/canary_rollout.bats
**Line:** 897:899
**Comment:**
	*Possible Bug: The test only checks that the note claims prompt changes do not autocut; it never exercises autocut, so it passes while `prompts/` changes still create candidates.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in tests/canary_rollout.bats: added a behavioral assertion (after line 902) that verifies .agents["solution-architect"].agent_ref_paths contains "scripts/" and "personas/" but does NOT contain "prompts/". This replaces the prior string-content-only check with an actual structural test of the override that suppresses autocut on prompt-only changes.

[ "$status" -eq 0 ]
# AC #2 (behavioral): agent_ref_paths overrides the global fleet default to exclude prompts/ so
# autocut's change detection never fires on a prompt-only change. scripts/ and personas/ are
# retained; prompts/ must be absent from the agent-level list.
run jq -e '.agents["solution-architect"].agent_ref_paths | (index("scripts/") != null) and (index("personas/") != null) and (index("prompts/") == null)' "$RINGS"
[ "$status" -eq 0 ]
}

@test "canary-rings.json: valid JSON + dev-lead host + ordered rings" {
run jq -e '.agents["dev-lead"].host == "petry-projects/.github-private"' "$RINGS"
[ "$status" -eq 0 ]
Expand Down