Update pr-review skill model lineup#35174
Conversation
Replace claude-sonnet-4.6 with claude-opus-4.7 and gemini-3-pro-preview with gpt-5.5 in the Phase 2 multi-model exploration list. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 35174Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 35174" |
🔍 Skill Validation Results✅ Static Checks PassedSkills checked: 15 | Agents checked: 3 Full validator output⏭️ LLM Evaluation: SkippedNo changed skills with eval tests found. |
|
/review |
|
✅ Expert Code Review completed successfully! |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Expert Code Review — PR #35174
Methodology: 3 independent reviewers with adversarial consensus
Findings
| # | Severity | Consensus | File | Lines | Finding |
|---|---|---|---|---|---|
| 1 | 🔴 CRITICAL | 3/3 reviewers | .github/skills/pr-review/SKILL.md |
53, 113 | claude-opus-4.7 is not in the platform's available model catalog. Attempt 2 of Phase 2 will fail at runtime. |
| 2 | 🔴 CRITICAL | 3/3 reviewers | .github/skills/pr-review/SKILL.md |
55, 117 | gpt-5.5 is not in the platform's available model catalog. Attempt 4 of Phase 2 will fail at runtime. |
| 3 | 🟢 MINOR | 3/3 reviewers | Other workflow files | — | claude-sonnet-4.6 still referenced in .github/workflows/shared/review-shared.md and .github/workflows/copilot-evaluate-tests.md, but these are separate workflows using it for their own purposes — not stale references from this PR. gemini-3-pro-preview is fully removed. No action needed. |
Details
Findings 1 & 2 — Unavailable model identifiers
The platform's task tool currently exposes these models: claude-opus-4.6, claude-opus-4.6-1m, claude-opus-4.5, claude-sonnet-4.6, claude-sonnet-4.5, claude-sonnet-4, claude-haiku-4.5, gpt-5.4, gpt-5.3-codex, gpt-5.2-codex, gpt-5.2, gpt-5.4-mini, gpt-5-mini, gpt-4.1.
Neither claude-opus-4.7 nor gpt-5.5 appears in this list. If these model IDs are not resolvable at runtime, 2 of 4 Phase 2 try-fix attempts will fail on every PR review, silently reducing fix exploration diversity by 50%.
If these models are expected to become available soon, consider gating the merge on their deployment. Otherwise, substitute with confirmed models.
Finding 3 — Other claude-sonnet-4.6 references
All 3 reviewers confirmed these are intentionally separate usages (agent models for different workflows), not stale references that should have been updated by this PR.
Internal Consistency ✅
The model config table (lines 50–55) and the Phase 2 launch checklist (lines 110–118) are consistent with each other after this change. No within-file discrepancies.
CI / Test Coverage
This PR modifies only a skill markdown file (no functional code). No CI tests are applicable or expected.
Generated by Expert Code Review for issue #35174 · ● 5.1M
| |-------|-------| | ||
| | 1 | `claude-opus-4.6` | | ||
| | 2 | `claude-sonnet-4.6` | | ||
| | 2 | `claude-opus-4.7` | |
There was a problem hiding this comment.
🔴 CRITICAL — Model not in available catalog (3/3 reviewers)
claude-opus-4.7 does not appear in the platform's current task-tool model catalog. The documented available models include claude-opus-4.6, claude-opus-4.6-1m, and claude-opus-4.5 — but not claude-opus-4.7.
If this model ID is not resolvable at runtime, Attempt 2 of every Phase 2 try-fix exploration will fail or be skipped, reducing fix diversity from 4 models to 3.
Recommendation: Confirm claude-opus-4.7 is a valid, deployed model identifier before merging. If not yet available, consider keeping claude-sonnet-4.6 or substituting with a confirmed model (e.g., claude-opus-4.6-1m or claude-opus-4.5).
| | 2 | `claude-opus-4.7` | | ||
| | 3 | `gpt-5.3-codex` | | ||
| | 4 | `gemini-3-pro-preview` | | ||
| | 4 | `gpt-5.5` | |
There was a problem hiding this comment.
🔴 CRITICAL — Model not in available catalog (3/3 reviewers)
gpt-5.5 does not appear in the platform's current task-tool model catalog. The documented available models include gpt-5.4, gpt-5.3-codex, gpt-5.2-codex, gpt-5.2, gpt-5.4-mini, gpt-5-mini, and gpt-4.1 — but not gpt-5.5.
If this model ID is not resolvable at runtime, Attempt 4 of every Phase 2 try-fix exploration will fail or be skipped, and the cross-pollination round will only cover 3 of 4 models.
Recommendation: Confirm gpt-5.5 is a valid, deployed model identifier before merging. If not yet available, consider keeping gemini-3-pro-preview or substituting with a confirmed model (e.g., gpt-5.4).
PureWeen
left a comment
PureWeen
left a comment
There was a problem hiding this comment.
Expert Code Review — PR #35174
Methodology: 3 independent reviewers with adversarial consensus.
Verdict: Safe to merge (1 advisory note) ✅
This PR updates a 4-line documentation table in .github/skills/pr-review/SKILL.md. The change is correct, internally consistent, and CI is green.
Findings
No bugs. All 3 reviewers (3/3) confirmed:
- ✅ Both new model IDs (
claude-opus-4.7,gpt-5.5) are present in the currenttask-tool model catalog. - ✅ The model table (lines 50–55) and the Phase 2 launch checklist (lines 111–117) are in sync after this change.
- ✅ No stale references to
gemini-3-pro-previewremain in the file. The remainingclaude-sonnet-4.6mentions in.github/workflows/shared/review-shared.mdand.github/workflows/copilot-evaluate-tests.md/.lock.ymlbelong to separate workflows (adversarial reviewer, test evaluator) and are correctly out of scope for this PR.
📝 An earlier automated review on this PR flagged the new model IDs as unavailable. That assessment was based on a stale catalog snapshot and is no longer accurate — both IDs are in the current catalog.
Advisory
| Before | After | |
|---|---|---|
| Position 1 | claude-opus-4.6 (Anthropic Opus) |
claude-opus-4.6 (Anthropic Opus) |
| Position 2 | claude-sonnet-4.6 (Anthropic Sonnet) |
claude-opus-4.7 (Anthropic Opus) |
| Position 3 | gpt-5.3-codex (OpenAI) |
gpt-5.3-codex (OpenAI) |
| Position 4 | gemini-3-pro-preview (Google) |
gpt-5.5 (OpenAI) |
| Vendors | 3 (Anthropic / OpenAI / Google) | 2 (Anthropic / OpenAI) |
| Anthropic capability tiers | 2 (Opus + Sonnet) | 1 (Opus only) |
The Phase 2 multi-model exploration is designed to surface divergent fix ideas. Two successive Opus versions tend to share reasoning posture and may converge on the same approach more often than Opus + Sonnet (different capability tier) or Opus + Gemini (different vendor) would.
The Gemini removal is forced — there are no Gemini IDs in the current catalog. The position-2 swap (Sonnet → Opus 4.7) is a deliberate choice that prioritizes "newer/stronger" over "more diverse."
Not blocking. If preserving adversarial diversity matters, one option is to keep claude-sonnet-4.6 at position 2 and place claude-opus-4.7 elsewhere (e.g., as part of cross-pollination). Otherwise a one-line note in the SKILL.md table about the explicit "strength-over-diversity" trade-off would help future maintainers understand the choice.
CI / Test Coverage
- All required checks pass.
maui-prand Build Analysis are skipped — normal for a docs-only diff. - No automated tests apply to skill prose.
Prior Reviews
- One existing automated
Expert Code Reviewcomment based on an older catalog snapshot; superseded by this review. - No unresolved human review threads.
Reviewed with multi-model adversarial consensus — Reviewers 1, 2, and 3 each evaluated the diff independently.
3 parallel reviewers (different models) ran against this PR; consensus findings: A) [3/3 ❌] pr-review/SKILL.md had stale model IDs (claude-sonnet-4.6 + gemini-3-pro-preview). Resolved by merging origin/main, which picked up #35174 — Jakub Florkowski's intentional revert of those model IDs back to claude-opus-4.7 + gpt-5.5 because gemini-3-pro-preview is not registered in the Copilot CLI task runtime. The pr-review/SKILL.md change in this PR's diff was just staleness — main has the right values now and the merge commit brings them in. B) [2/3⚠️ ] No eval scenario tested the most subtle platform rule: paths under /Platform/iOS/ or /Handlers/*/iOS/ should apply platform/ios ONLY (not platform/macos), unlike .ios.cs file extension which applies BOTH. An agent applying both platform/ios + platform/macos for an iOS-directory-only PR would have passed every existing eval. Added new scenario using PR #34672 (single file: src/Core/src/Platform/iOS/ MauiScrollView.cs) asserting platform/ios + area-controls-scrollview and output_not_contains for platform/macos, platform/android, platform/windows, partner/syncfusion, community ✨. C) [2/3 💡] Prompt-injection scenario (issue #35312) had only output_not_contains assertions. An agent that completely noops or returns empty output would pass. Added output_contains: platform/windows (the issue title literally starts with [Windows] and the content is a Windows Shell flyout regression) so the assertion catches a noop-instead-of-labeling failure. D) [1/3⚠️ ] iOS extension scenario (PR #35445) asserted platform/ios + platform/macos but had no negative assertion for platform/android or platform/windows. Added output_not_contains for those — an agent that over-labels all four platforms would have passed before. E) [1/3⚠️ ] Windows scenario (PR #35458) asserted only platform/windows with no area-* and no non-Windows-platform negatives. Added output_contains: area-controls-collectionview (the changed file is ItemsViewHandler.Windows.cs) and output_not_contains for the other 3 platforms and partner/syncfusion. Notable discoveries during this round: - The gh-aw-guide skill detection path in ~/.agents/skills/generic-adversarial-pr-reviewer/SKILL.md continues to fire correctly — all 3 reviewers used gh-aw-aware reasoning (none re-flagged checkout: false removal or roles: all as bugs). - Eval scenario count increased from 20 → 21 (new iOS-dir-only scenario). - lock.yml unchanged (no workflow.md frontmatter changes); only eval.yaml modified in this commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
> [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! Updates the Phase 2 multi-model exploration list in the `pr-review` skill: | Order | Before | After | |-------|--------|-------| | 1 | claude-opus-4.6 | claude-opus-4.6 (unchanged) | | 2 | **claude-sonnet-4.6** | **claude-opus-4.7** | | 3 | gpt-5.3-codex | gpt-5.3-codex (unchanged) | | 4 | **gemini-3-pro-preview** | **gpt-5.5** | Updated in both the model config table and the Phase 2 launch checklist in `.github/skills/pr-review/SKILL.md`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
> [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! Updates the Phase 2 multi-model exploration list in the `pr-review` skill: | Order | Before | After | |-------|--------|-------| | 1 | claude-opus-4.6 | claude-opus-4.6 (unchanged) | | 2 | **claude-sonnet-4.6** | **claude-opus-4.7** | | 3 | gpt-5.3-codex | gpt-5.3-codex (unchanged) | | 4 | **gemini-3-pro-preview** | **gpt-5.5** | Updated in both the model config table and the Phase 2 launch checklist in `.github/skills/pr-review/SKILL.md`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Updates the Phase 2 multi-model exploration list in the
pr-reviewskill:Updated in both the model config table and the Phase 2 launch checklist in
.github/skills/pr-review/SKILL.md.