test(skills): add issue-derived Pythonic code evals - #1099
Conversation
Signed-off-by: phernandez <paul@basicmachines.co>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: befe112be0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f83bfd9c98
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8fca87a5f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 081abd48c8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Why
PR #1095 introduced the
/pythonic-codeskill, but the skill had no repeatable evaluation surface. Real cleanup issues #1096, #1097, and #1098 provide representative architecture, persistence-invariant, and abstraction-simplification cases without relying on toy Python examples.What Changed
justfilefor listing, validating, dry-running, and running paired trials.Protocolfor replaceable behavior, not property-only internal result shapes.Implementation Details
The runner copies only the selected fixture files into a fresh temporary workspace. Skill-guided trials receive
SKILL.mdbut not the eval definitions, assertions, expected outputs, runner, or prior results.The first fair-context run exposed a concrete weakness: the skill-guided #1098 patch retained two property-only result protocols and scored 3/5 while the baseline scored 5/5. After adding the focused protocol rule, the unchanged #1098 retest scored 5/5; its concurrent baseline scored 4/5 because it introduced an
AcceptedNotePreparerclass instead of direct module-level behavior.Using the latest result for each unchanged case, the revised skill scored 15/15 versus 14/15 for the baseline. These are directional single-run results; the code-shape difference is the useful signal, while timing and token counts remain stochastic.
Testing
Passed:
cd .agents/skills/pythonic-code && just validatePYTHONPATH=evals/files uv run pytest -q evals/files/test_accepted_snapshot.py evals/files/test_accepted_preparation.py— 5 passeduv run python /Users/phernandez/.codex/skills/.system/skill-creator/scripts/quick_validate.py .agents/skills/pythonic-codejust package-checkRisks / Follow-ups
Related to #1096, #1097, and #1098.