Repository navigation
[Phase 3] Add a model-version-pin lint check wired into CI #1212
Copy link
Copy link
Open
Labels
Description
Story
As a CI maintainer,
I want add a deterministic lint that flags claude-(opus|sonnet|haiku|fable)-[0-9] version pins in code, YAML values and markdown defaults - skipping # model-pin-ok: lines and the allowed-exception files - and wire it into ci.yml,
so that the family-not-version standard is enforced automatically and future version pins are caught in CI instead of drifting across the fleet.
Acceptance Criteria
- A new deterministic lint (pure shell + jq, following the scripts/agents-md-lint.sh pattern) flags
claude-(opus|sonnet|haiku|fable)-[0-9]version pins, skipping any line carrying a# model-pin-ok:marker and the allowed-exception files (resolver, price data, recorded fixtures/eval sets/baselines, fixed eval judge). - The check inspects YAML string values (e.g.
ANTHROPIC_MODEL: claude-opus-4-6), not only shell/code identifiers, so it catches the env-var-style pin called out in enhancement comment risk Add stacked PR strategy and Epic-level workflow guidance #5. - The check is wired into .github/workflows/ci.yml's
lintjob as a new sub-step withif: always()(mirroringrun: bash scripts/check-cron-timing.sh). It runs in informational mode emitting ::warning:: annotations on introduction (per the agents-md-selfcheck precedent); promotion to--mode failingis a follow-up decision, taken only once the fleet is confirmed pin-free, and is NOT added to branch protection while informational. - Unit tests (bats in tests/, run via scripts/run-bats.sh) cover: a pinned id fails, a family alias passes, a
# model-pin-ok:exception passes, and a YAML env-value pin fails. - The regex and exception skip-list match [Phase 3] Standard: name a model family, never pin a model version — replace every hard-coded Claude model id and lint for it .github-private#1979's check so the two repos stay consistent.
- The new script passes
shellcheck --severity=warning -x(the ShellCheck CI job lints scripts/**/*.sh).
Tasks / Subtasks
- Write scripts model-pin lint modelled on scripts/agents-md-lint.sh: deterministic, sourceable pure helpers, TSV findings, informational|failing mode. (AC: Addressing PR comments #1, Add multi-agent isolation strategy using git worktrees #2, Add coding standards & principles section to AGENTS.md #3, feat: add Structured Logging and CQRS standards #6)
- Implement YAML-value scanning so
ANTHROPIC_MODEL: claude-opus-4-6-style pins are caught, plus the# model-pin-ok:and exception-file skips. (AC: Addressing PR comments #1, Add multi-agent isolation strategy using git worktrees #2) - Wire an
if: always()sub-step into ci.yml's Lint job invoking the script (informational to start). (AC: Add coding standards & principles section to AGENTS.md #3) - Add bats tests in tests/ (pinned id, family alias, model-pin-ok exception, YAML env-value pin) runnable via run-bats.sh. (AC: Add workflow, environment, and orchestration guidance #4)
- Align the regex/skip-list with .github-private#1979 and confirm shellcheck passes. (AC: Add stacked PR strategy and Epic-level workflow guidance #5, feat: add Structured Logging and CQRS standards #6)
Dev Notes
- Model the script on scripts/agents-md-lint.sh: data-driven,
source-able side-effect-free helpers (so bats can exercise them), deterministic TSV findings, and an informational (exit 0, ::warning::) vs failing (--mode failing) toggle. Follow run-bats.sh's non-fatal-signal convention for informational mode. - Wiring precedent in .github/workflows/ci.yml: the
lintjob runs sibling sub-steps such asrun: bash scripts/check-cron-timing.shwithif: always()so every lint issue surfaces in one run; theshellcheckjob runsshellcheck --severity=warning -x scripts/**/*.sh; theagents-md-selfcheckjob is the informational-then-Phase-4-failing precedent (do NOT make this a required branch-protection check while informational). - Allowed-exception files come straight from the Story 1 standard: the resolver, price data keyed by real ids, recorded fixtures/eval sets/baselines, and the fixed eval judge. Prefer a small skip-list/data file over hardcoding, mirroring scripts/lib/agents-md-rules.json's data-driven approach.
- [Phase 3] Standard: name a model family, never pin a model version — replace every hard-coded Claude model id and lint for it .github-private#1979 (under epic #1895) is the sibling implementation - align the regex
claude-(opus|sonnet|haiku|fable)-[0-9]and the skip-list with it; it is cross-repo so it is an untracked prerequisite, not a blocked_by edge. - Land AFTER Stories 2 and 3 so the known pins (reusable default + ci-standards section 9) are already removed and the new lint does not red-flag the very files being fixed. Start informational for the same reason (a pin missed elsewhere should not block this PR).
- Testing standard: bats tests live in tests/ and run via scripts/run-bats.sh; the script itself must pass shellcheck.
Project Structure Notes
New model-pin lint script + optional scripts/lib data file + tests/*.bats, plus one ci.yml Lint-job step. Follows the repo's existing linter+bats+ci-wiring convention (agents-md-lint.sh, check-cron-timing.sh).
References
- https://github.com/petry-projects/.github/blob/main/scripts/agents-md-lint.sh
- https://github.com/petry-projects/.github/blob/main/.github/workflows/ci.yml
- https://github.com/petry-projects/.github/blob/main/scripts/check-cron-timing.sh
- discussion Standard Proposal: name a model family, never pin a model version #1199
Likely target surface
petry-projects/.github: new model-pin lint script + tests/*.bats + ci.yml Lint job step
Story prepared by the BMAD Scrum Master (Bob) for epic #1208. Status: ready-for-dev.
Reactions are currently unavailable
Activity