Skip to content

[Phase 3] Add a model-version-pin lint check wired into CI #1212

Open

Description

@don-petry

Story

As a CI maintainer,
I want add a deterministic lint that flags claude-(opus|sonnet|haiku|fable)-[0-9] version pins in code, YAML values and markdown defaults - skipping # model-pin-ok: lines and the allowed-exception files - and wire it into ci.yml,
so that the family-not-version standard is enforced automatically and future version pins are caught in CI instead of drifting across the fleet.

Acceptance Criteria

  1. A new deterministic lint (pure shell + jq, following the scripts/agents-md-lint.sh pattern) flags claude-(opus|sonnet|haiku|fable)-[0-9] version pins, skipping any line carrying a # model-pin-ok: marker and the allowed-exception files (resolver, price data, recorded fixtures/eval sets/baselines, fixed eval judge).
  2. The check inspects YAML string values (e.g. ANTHROPIC_MODEL: claude-opus-4-6), not only shell/code identifiers, so it catches the env-var-style pin called out in enhancement comment risk Add stacked PR strategy and Epic-level workflow guidance #5.
  3. The check is wired into .github/workflows/ci.yml's lint job as a new sub-step with if: always() (mirroring run: bash scripts/check-cron-timing.sh). It runs in informational mode emitting ::warning:: annotations on introduction (per the agents-md-selfcheck precedent); promotion to --mode failing is a follow-up decision, taken only once the fleet is confirmed pin-free, and is NOT added to branch protection while informational.
  4. Unit tests (bats in tests/, run via scripts/run-bats.sh) cover: a pinned id fails, a family alias passes, a # model-pin-ok: exception passes, and a YAML env-value pin fails.
  5. The regex and exception skip-list match [Phase 3] Standard: name a model family, never pin a model version — replace every hard-coded Claude model id and lint for it .github-private#1979's check so the two repos stay consistent.
  6. The new script passes shellcheck --severity=warning -x (the ShellCheck CI job lints scripts/**/*.sh).

Tasks / Subtasks

Dev Notes

  • Model the script on scripts/agents-md-lint.sh: data-driven, source-able side-effect-free helpers (so bats can exercise them), deterministic TSV findings, and an informational (exit 0, ::warning::) vs failing (--mode failing) toggle. Follow run-bats.sh's non-fatal-signal convention for informational mode.
  • Wiring precedent in .github/workflows/ci.yml: the lint job runs sibling sub-steps such as run: bash scripts/check-cron-timing.sh with if: always() so every lint issue surfaces in one run; the shellcheck job runs shellcheck --severity=warning -x scripts/**/*.sh; the agents-md-selfcheck job is the informational-then-Phase-4-failing precedent (do NOT make this a required branch-protection check while informational).
  • Allowed-exception files come straight from the Story 1 standard: the resolver, price data keyed by real ids, recorded fixtures/eval sets/baselines, and the fixed eval judge. Prefer a small skip-list/data file over hardcoding, mirroring scripts/lib/agents-md-rules.json's data-driven approach.
  • [Phase 3] Standard: name a model family, never pin a model version — replace every hard-coded Claude model id and lint for it .github-private#1979 (under epic #1895) is the sibling implementation - align the regex claude-(opus|sonnet|haiku|fable)-[0-9] and the skip-list with it; it is cross-repo so it is an untracked prerequisite, not a blocked_by edge.
  • Land AFTER Stories 2 and 3 so the known pins (reusable default + ci-standards section 9) are already removed and the new lint does not red-flag the very files being fixed. Start informational for the same reason (a pin missed elsewhere should not block this PR).
  • Testing standard: bats tests live in tests/ and run via scripts/run-bats.sh; the script itself must pass shellcheck.

Project Structure Notes

New model-pin lint script + optional scripts/lib data file + tests/*.bats, plus one ci.yml Lint-job step. Follows the repo's existing linter+bats+ci-wiring convention (agents-md-lint.sh, check-cron-timing.sh).

References

Likely target surface

  • petry-projects/.github: new model-pin lint script + tests/*.bats + ci.yml Lint job step

Story prepared by the BMAD Scrum Master (Bob) for epic #1208. Status: ready-for-dev.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions