Skip to content

Add scheduled authority-source drift checking - #47

Merged
akoita merged 1 commit into
mainfrom
claude/festive-shannon-dwsjko
Oct 20, 2026
Merged

akoita merged 1 commit into
mainfrom
claude/festive-shannon-dwsjko

Conversation

@akoita

@akoita akoita commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Summary

Skills pin tool releases and cite primary sources that go stale without any signal. This adds the first two stages of a source-verification pipeline, as maintainer-side CI rather than a client hook.

  • Manifest — .github/authority-sources.json registers 23 sources with the canonical file that depends on each, its publisher, and a verified_on date. Coverage starts with the whole security-scan skill (17 pinned releases, 3 URLs) and the Google-sourced security references (Chrome Security blog, Chromium AI-bugs FAQ, GTIG supply-chain guidance).
  • Coverage contract — tests/test_authority_sources.py fails a PR that adds an unregistered URL or # vX.Y.Z pin to a tracked file, cites a generated mirror instead of the canonical file, or leaves an entry whose marker/URL the file no longer contains. Runs offline under the existing unittest job.
  • Checker — .github/scripts/check_sources.py (stdlib only, no model) compares pins with the latest GitHub/PyPI/npm/RubyGems release, flags a pin ahead of any release (catches fabricated versions), probes cited pages for 404s and redirects, and marks URL claims due for review after review_after_days (default 90). Exits 0 clean / 1 attention / 2 invalid manifest; --json output is the input contract for a future agent stage.
  • Workflow — source-drift runs weekly (Mon 06:17 UTC) and on workflow_dispatch, and keeps one Authority source drift report issue current (create, update with a comment when findings change, close when clean). It never edits a skill.
  • Docs — "Authority sources" section in docs/contributing.md; ADR 0003 records the design and the proposed stages 3 (agent review of flagged entries, using Google's Developer Knowledge MCP for publisher: google sources) and 4 (monthly discovery notes).

Security notes

  • The job that fetches third-party content has contents: read only; the issues: write job fetches nothing and receives the report through an env var, never script interpolation.
  • The GitHub token is attached as an unredirected header, sent only to api.github.com.
  • Registry version strings must match a strict pattern and redirect targets are sanitized before reaching the issue body.
  • actions/checkout is pinned to a commit SHA with persist-credentials: false; a test enforces SHA pinning for this workflow.

Expected first report

verified_on dates are seeded from dates the skills already record (2026-07-29 toolchain, 2026-08-03 Google research), not re-verified. A local run confirmed checkov 3.3.8→3.3.30, schemathesis 4.24.3→4.32.2, eslint-plugin-security 4.0.1→4.2.0, and brakeman 8.0.5→8.1.0. GitHub API lookups were blocked in the authoring sandbox; git tags suggest trivy, osv-scanner, syft, grype, trufflehog, gosec, nuclei, and AFL++ are also behind. Fixing that drift is left for a follow-up, since each bump needs flag re-verification and a package version bump.

Test plan

  • python -m unittest discover -s tests — 110 tests pass
  • sync_skills.py --check and sync_plugin_adapters.py --check clean
  • No package files changed, so no version bump required
  • check_sources.py --validate-only passes; live run produces the report above
  • After merge: run source-drift via workflow_dispatch and confirm the tracking issue opens

Generated by Claude Code

Skills pin tool releases and cite primary sources that go stale silently.
Register those sources in .github/authority-sources.json, starting with the
security-scan skill and the Google-sourced security references, and check
them weekly:

- check_sources.py compares pins with the latest GitHub, PyPI, npm, and
  RubyGems releases, probes cited pages for removal or relocation, and marks
  claims due for review after their interval. Stdlib only, no model.
- tests/test_authority_sources.py fails a pull request that adds an
  unregistered URL or pinned version to a tracked file, or leaves an entry
  whose marker or URL the file no longer contains.
- The source-drift workflow keeps one tracking issue current. The job that
  fetches third-party content cannot write; the issue-writing job fetches
  nothing. Actions are pinned to commit SHAs.

ADR 0003 records the design and the proposed agent-review stage, which
would use Google's Developer Knowledge MCP for Google-published sources.
verified_on dates are seeded from the dates the skills already record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BCqiHZzRJbY1VLSgkm14k
@akoita
akoita merged commit edb824d into main Oct 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants