Skip to content

[Phase 3] Add a model-version-pin lint check wired into CI #1212

Description

@don-petry

Story

As a CI maintainer,
I want add a deterministic lint that flags claude-(opus|sonnet|haiku|fable)-[0-9] version pins in code, YAML values and markdown defaults - skipping # model-pin-ok: lines and the allowed-exception files - and wire it into ci.yml,
so that the family-not-version standard is enforced automatically and future version pins are caught in CI instead of drifting across the fleet.

Acceptance Criteria

  1. A new deterministic lint (pure shell + jq, following the scripts/agents-md-lint.sh pattern) flags claude-(opus|sonnet|haiku|fable)-[0-9] version pins, skipping any line carrying a # model-pin-ok: marker and the allowed-exception files (resolver, price data, recorded fixtures/eval sets/baselines, fixed eval judge).
  2. The check inspects YAML string values (e.g. ANTHROPIC_MODEL: claude-opus-4-6), not only shell/code identifiers, so it catches the env-var-style pin called out in enhancement comment risk Add stacked PR strategy and Epic-level workflow guidance #5.
  3. The check is wired into .github/workflows/ci.yml's lint job as a new sub-step with if: always() (mirroring run: bash scripts/check-cron-timing.sh). It runs in informational mode emitting ::warning:: annotations on introduction (per the agents-md-selfcheck precedent); promotion to --mode failing is a follow-up decision, taken only once the fleet is confirmed pin-free, and is NOT added to branch protection while informational.
  4. Unit tests (bats in tests/, run via scripts/run-bats.sh) cover: a pinned id fails, a family alias passes, a # model-pin-ok: exception passes, and a YAML env-value pin fails.
  5. The regex and exception skip-list match [Phase 3] Standard: name a model family, never pin a model version — replace every hard-coded Claude model id and lint for it .github-private#1979's check so the two repos stay consistent.
  6. The new script passes shellcheck --severity=warning -x (the ShellCheck CI job lints scripts/**/*.sh).

Tasks / Subtasks

Dev Notes

  • Model the script on scripts/agents-md-lint.sh: data-driven, source-able side-effect-free helpers (so bats can exercise them), deterministic TSV findings, and an informational (exit 0, ::warning::) vs failing (--mode failing) toggle. Follow run-bats.sh's non-fatal-signal convention for informational mode.
  • Wiring precedent in .github/workflows/ci.yml: the lint job runs sibling sub-steps such as run: bash scripts/check-cron-timing.sh with if: always() so every lint issue surfaces in one run; the shellcheck job runs shellcheck --severity=warning -x scripts/**/*.sh; the agents-md-selfcheck job is the informational-then-Phase-4-failing precedent (do NOT make this a required branch-protection check while informational).
  • Allowed-exception files come straight from the Story 1 standard: the resolver, price data keyed by real ids, recorded fixtures/eval sets/baselines, and the fixed eval judge. Prefer a small skip-list/data file over hardcoding, mirroring scripts/lib/agents-md-rules.json's data-driven approach.
  • [Phase 3] Standard: name a model family, never pin a model version — replace every hard-coded Claude model id and lint for it .github-private#1979 (under epic #1895) is the sibling implementation - align the regex claude-(opus|sonnet|haiku|fable)-[0-9] and the skip-list with it; it is cross-repo so it is an untracked prerequisite, not a blocked_by edge.
  • Land AFTER Stories 2 and 3 so the known pins (reusable default + ci-standards section 9) are already removed and the new lint does not red-flag the very files being fixed. Start informational for the same reason (a pin missed elsewhere should not block this PR).
  • Testing standard: bats tests live in tests/ and run via scripts/run-bats.sh; the script itself must pass shellcheck.

Project Structure Notes

New model-pin lint script + optional scripts/lib data file + tests/*.bats, plus one ci.yml Lint-job step. Follows the repo's existing linter+bats+ci-wiring convention (agents-md-lint.sh, check-cron-timing.sh).

References

Likely target surface

  • petry-projects/.github: new model-pin lint script + tests/*.bats + ci.yml Lint job step

Story prepared by the BMAD Scrum Master (Bob) for epic #1208. Status: ready-for-dev.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions