Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion plugins/skill-quality/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
"name": "skill-quality",
"version": "0.5.0",
"version": "0.6.0",
"description": "Skill-authoring QA tooling: a static contract checker that runs seventeen deterministic checks over a Claude Code skill (frontmatter, listing-budget cap, trigger-keyword preservation, line caps, broken internal refs, markdownlint, gotchas surface, evals presence) and a bundled evals.json schema for validation. Runs against any repo's skills directory via the convention-resolution ladder — no baked layout.",
"author": {
"name": "Melodic Software",
Expand Down
14 changes: 14 additions & 0 deletions plugins/skill-quality/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,20 @@
All notable changes to the `skill-quality` plugin are documented here. Format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning.

## [0.6.0]

### Added

- **Check 1 now enforces that frontmatter `name` matches the skill directory name.**
`docs/PLUGIN-PHILOSOPHY.md` has always required it, but nothing verified it — check 1 asserted
only that `name:` was present and non-empty. The directory name is what Claude Code namespaces the
skill by, so a divergent frontmatter `name` silently relocates the invocation the doctrine says
the skill has, and because the slash-command picker labels rows by the resolved leaf name the
drift never surfaced in the listing either. Lands as a deterministic FAIL rather than a warning:
the whole catalog (144 skills) already conforms, so there is no debt to grandfather and no
baseline file. A quoted value is unquoted before comparison, and an absent `name` still reports
only the existing missing-`name` failure rather than a spurious second one.

## [0.5.0]

### Changed
Expand Down
28 changes: 26 additions & 2 deletions plugins/skill-quality/scripts/check-skill.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
# (e.g. HEAD^ or a merge-base).
#
# Checks:
# 1. Frontmatter parses; name + description present
# 1. Frontmatter parses; name matches dir; description present
# 2. description + when_to_use <= 1536 chars (listing-truncation guard)
# 3. Trigger-keyword preservation vs the base ref (skipped for new skills)
# 4. SKILL.md < 500 lines (hard cap)
Expand Down Expand Up @@ -150,14 +150,38 @@ if [[ ! -f "$SKILL_MD" ]]; then
exit 1
fi

# --- Check 1: frontmatter parses; name + description present ---------------
# --- Check 1: frontmatter parses; name matches dir; description present ----

FRONTMATTER="$(skill_frontmatter::extract <"$SKILL_MD")"
if [[ -z "$FRONTMATTER" ]]; then
err "no YAML frontmatter block found (expected content between two '---' fences)"
else
grep -qE '^name:[[:space:]]*\S' <<<"$FRONTMATTER" || err "frontmatter missing 'name:'"
grep -qE '^description:[[:space:]]*\S' <<<"$FRONTMATTER" || err "frontmatter missing 'description:'"

# The directory name is what Claude Code namespaces the skill by, so a
# divergent frontmatter name silently relocates the invocation the doctrine
# says the skill has — and the picker labels rows by the resolved leaf name,
# so the drift never surfaces in the listing either.
RAW_NAME="$(skill_frontmatter::field name <<<"$FRONTMATTER")"
# A trailing `# comment` is legal on a YAML scalar and is not part of the
# value. Skill names are kebab-case per the Agent Skills spec, so a '#' can
# never belong to the name itself — strip from the first whitespace-then-hash,
# before unquoting, so a quoted name with a trailing comment also resolves.
RAW_NAME="${RAW_NAME%%[[:space:]]#*}"
RAW_NAME="${RAW_NAME%"${RAW_NAME##*[![:space:]]}"}"
CUR_NAME="$(skill_frontmatter::strip_quotes "$RAW_NAME")"
Comment thread
kyle-sexton marked this conversation as resolved.
# Constrain the accepted syntax rather than reimplementing a YAML decoder in
# bash: the Agent Skills spec restricts a name to lowercase alphanumerics and
# hyphens, so anything else (an escape sequence like "\x2d", whitespace, an
# unresolved quote) is a name defect in its own right. Reporting it as one
# keeps the directory comparison below working on literal text, and stops a
# decodable-but-undecoded scalar from surfacing as a confusing mismatch.
if [[ -n "$CUR_NAME" && ! "$CUR_NAME" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]; then
err "frontmatter name '$CUR_NAME' is not kebab-case ([a-z0-9] and hyphens, per the Agent Skills spec)"
elif [[ -n "$CUR_NAME" && "$CUR_NAME" != "$SKILL_NAME" ]]; then
err "frontmatter name '$CUR_NAME' does not match skill directory '$SKILL_NAME'"
fi
fi

# --- Check 2: description + when_to_use <= DESC_CHAR_CAP chars --------------
Expand Down
116 changes: 116 additions & 0 deletions plugins/skill-quality/scripts/check-skill.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,122 @@ else
fail "trigger on a less-indented block line should be tracked (rc=$rc): $out"
fi

# 14. Frontmatter name diverging from the skill directory fails (check 1). The
# directory name is what the skill is namespaced by, so a mismatch silently
# relocates the invocation the doctrine says the skill has.
make_skill misnamed-skill '---
name: some-other-name
description: "Name does not match its directory. Use when: '"'"'checking the name gate'"'"'."
---

## Purpose

Fixture whose frontmatter name diverges from its directory name.

## Gotchas

None known.
'
out="$(run misnamed-skill 2>&1)"
rc=$?
if [[ $rc -eq 1 ]] && grep -q "does not match skill directory 'misnamed-skill'" <<<"$out"; then
pass "frontmatter name mismatching the directory fails"
else
fail "name/directory mismatch should fail (rc=$rc): $out"
fi

# 15. A matching name does not trip the gate — guards the false-positive
# direction, and proves a quoted value reaches the comparison unquoted.
make_skill quoted-name '---
name: "quoted-name"
description: "Quoted name matching its directory. Use when: '"'"'checking quoted names'"'"'."
---

## Purpose

Fixture whose frontmatter name is quoted but matches its directory.

## Gotchas

None known.
'
out="$(run quoted-name 2>&1)"
rc=$?
if [[ $rc -eq 0 ]] && ! grep -q 'does not match skill directory' <<<"$out"; then
pass "quoted name matching the directory does not trip the gate"
else
fail "quoted matching name should not trip the gate (rc=$rc): $out"
fi

# 16. A trailing YAML comment is not part of the scalar, so a correctly named
# skill carrying one must not trip the gate (false-positive direction).
make_skill commented-name '---
name: commented-name # kept for the migration note
description: "Name carries a trailing YAML comment. Use when: '"'"'checking comment stripping'"'"'."
---

## Purpose

Fixture whose frontmatter name carries a legal trailing comment.

## Gotchas

None known.
'
out="$(run commented-name 2>&1)"
rc=$?
if [[ $rc -eq 0 ]] && ! grep -q 'does not match skill directory' <<<"$out"; then
pass "trailing YAML comment on name does not trip the gate"
else
fail "commented name should not trip the gate (rc=$rc): $out"
fi

# 16b. A YAML escape sequence is reported as the name defect it is, rather than
# surfacing as a confusing directory mismatch. Constraining the accepted
# syntax is the alternative to decoding YAML in bash.
make_skill escaped-name '---
name: "escaped\x2dname"
description: "Name carries a YAML escape. Use when: '"'"'checking the charset gate'"'"'."
---

## Purpose

Fixture whose frontmatter name uses a YAML escape sequence.

## Gotchas

None known.
'
out="$(run escaped-name 2>&1)"
rc=$?
if [[ $rc -eq 1 ]] && grep -q 'is not kebab-case' <<<"$out"; then
pass "a YAML escape in the name reports as a charset defect, not a mismatch"
else
fail "escaped name should fail the charset gate (rc=$rc): $out"
fi

# 17. Comment stripping must not mask a real mismatch.
make_skill commented-bad '---
name: wrong-name # with a comment too
description: "Wrong name plus a comment. Use when: '"'"'checking comment stripping'"'"'."
---

## Purpose

Fixture that is both misnamed and commented.

## Gotchas

None known.
'
out="$(run commented-bad 2>&1)"
rc=$?
if [[ $rc -eq 1 ]] && grep -q "frontmatter name 'wrong-name' does not match" <<<"$out"; then
pass "comment stripping does not mask a real mismatch"
else
fail "commented misnamed skill should still fail (rc=$rc): $out"
fi

if [[ $fails -ne 0 ]]; then
printf '%d assertion(s) failed\n' "$fails" >&2
exit 1
Expand Down
Loading