Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
"actionlint@melodic-software": true,
"adhd@melodic-software": true,
"ai-briefing@melodic-software": true,
"ai-slop@melodic-software": true,
"architecture@melodic-software": true,
"autonomy@melodic-software": true,
"bash-format@melodic-software": true,
Expand All @@ -36,6 +37,7 @@
"code-tidying@melodic-software": true,
"codebase-health@melodic-software": true,
"computer-use@melodic-software": true,
"context-budget@melodic-software": true,
"context-guard@melodic-software": true,
"context7@melodic-software": true,
"coupling@melodic-software": true,
Expand All @@ -56,6 +58,7 @@
"go-format@melodic-software": true,
"guardrails@melodic-software": true,
"implementation@melodic-software": true,
"improvement@melodic-software": true,
"kindle-dedrm@melodic-software": true,
"knowledge@melodic-software": true,
"machine-health@melodic-software": true,
Expand Down
26 changes: 26 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -731,6 +731,31 @@ jobs:
- name: Check every catalog entry resolves to a present, name-matching plugin manifest
run: scripts/check-plugin-manifest-presence.sh

# docs/CLOUD-SESSIONS.md promises `enabledPlugins` "turns on the whole
# catalog, so this repo dogfoods everything it publishes"; nothing enforced
# it, and ai-slop (#2892), context-budget (#2932) and improvement (#2985)
# each reached main catalogued but never enabled. The failure is silent by
# construction: .claude/cloud-bootstrap.sh computes its install set from that
# same enabledPlugins map, so a session comes up green with the plugin's
# skills simply absent and nothing naming what is missing. The consumer-side
# detector cannot cover this — check-plugin-drift.sh resolves marketplaces
# through `source.repo` and SKIPs one that declares none, which is exactly
# this repo's relative `directory` source. Repo-wide and static, so it always
# runs. The self-test runs first so a broken detector cannot mask a
# regression behind a green gate.
plugin-catalog-enablement-gate:
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Test the plugin-catalog-enablement gate
run: bash scripts/check-plugin-catalog-enablement.test.sh
- name: Check every catalogued plugin carries an enabledPlugins key
run: scripts/check-plugin-catalog-enablement.sh

# Every file under a skill's evals/fixtures/ must be consumed by a grader — an
# eval files[] entry or a test assertion — so an ungraded suite cannot sit in
# the tree reading as tested. The check is repo-wide and static; existing debt
Expand Down Expand Up @@ -1471,6 +1496,7 @@ jobs:
- silent-skip-gate
- stale-base-overlap-gate
- plugin-manifest-presence-gate
- plugin-catalog-enablement-gate
- orphaned-fixture-gate
- fixture-git-isolation-gate
- fleet-finding-test-coverage-gate
Expand Down
15 changes: 14 additions & 1 deletion docs/CLOUD-SESSIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,8 +274,21 @@ enables; the cloud bootstrap installs (see
this set wholesale. The cloud bootstrap provisions every tool the format/lint-on-edit hooks
(`markdown-format`, `bash-format`, `biome-format`, `typos-format`, `actionlint`,
`eol-normalizer`) shell out to.
- The `plugin-catalog-enablement-gate` CI lane holds that whole-catalog claim to the file, in both
directions: every `.claude-plugin/marketplace.json` entry must carry an `enabledPlugins` key, and
every key for this marketplace must name a catalogued plugin. It also checks that
`cloud-bootstrap.sh`'s hardcoded `marketplace_name` still names the marketplace the settings file
declares — the bootstrap selects what it installs with `endswith("@" + $n)`, so a rename that
updated the settings and the catalog but not that constant would leave its install set empty
while the parity lane stayed green over it. It exists because the claim was
prose for three plugin releases that shipped catalogued but never enabled — a silent failure,
since the bootstrap computes its install set from the same map and a session simply comes up
without those skills. `claude-config`'s `check-plugin-drift.sh` cannot cover it: that detector
resolves each marketplace through `source.repo` and records SKIP for one declaring none, which
is precisely this repo's relative `directory` source.
- Entries are sorted alphabetically, one per line, so a single plugin can be flipped to `false`
without disturbing the rest. Two entries carry required `userConfig` credentials that are unset
without disturbing the rest — a state the gate accepts, since an explicit `false` is a recorded
decision where an absent key is drift. Two entries carry required `userConfig` credentials that are unset
here — `miro` (`miro_api_token`) and `dometrain` (`dometrain_api_key`) — so their bundled MCP
servers exit at startup until configured; set them with `/plugin configure`, or flip those two
to `false` if a session shouldn't try.
Expand Down
28 changes: 26 additions & 2 deletions scripts/affected-tests.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -577,9 +577,33 @@ done
# nothing about .github/*. That is not hypothetical — .github/workflows/ci.yml
# is named by two suites and reaches exit 0 through R3, so it cannot serve as
# this probe. Discovered by glob for the R3 reason given above.
mapfile -t wf_yaml < <(cd "$REPO_ROOT" && git ls-files '.github/*.yaml' | head -1)
#
# The candidate is additionally FILTERED to one that no grepped-language file
# names at all, rather than assuming the sole `.github/*.yaml` qualifies. That
# assumption held until a suite acquired a real transitive claim on it:
# .claude/cloud-bootstrap.sh's pin comment names .github/actionlint.yaml, and
# scripts/check-plugin-catalog-enablement.sh reads cloud-bootstrap.sh, so the
# walk now runs actionlint.yaml -> cloud-bootstrap.sh -> that gate's suite and
# reaches exit 0 through R3 exactly the way ci.yml does. Both edges are real and
# neither should be severed to keep a probe convenient, so the probe moves
# instead — the same resolution the ci.yml sentence above records.
#
# The filter is an INDEPENDENT oracle (a direct git grep for the basename), not
# a call to affected-tests.sh: picking the probe with the tool under test would
# make the assertion below tautological. One incoming reference anywhere is
# enough to disqualify a candidate, because R3's first level would then have
# somewhere to go.
mapfile -t wf_candidates < <(cd "$REPO_ROOT" && git ls-files '.github/*.yaml' '.github/*.yml')
wf_yaml=()
for c in ${wf_candidates[@]+"${wf_candidates[@]}"}; do
if ! (cd "$REPO_ROOT" && git grep -q -F -- "${c##*/}" \
-- '*.sh' '*.bash' '*.js' '*.mjs' '*.cjs' '*.py' '*.ps1' '*.psm1') 2>/dev/null; then
wf_yaml=("$c")
break
fi
done
if [[ ${#wf_yaml[@]} -eq 0 ]]; then
fail "no .github YAML found to probe — the case below would be vacuous"
fail "no unreferenced .github YAML found to probe — the case below would be vacuous"
fi
for y in ${wf_yaml[@]+"${wf_yaml[@]}"}; do
out="$(cd "$REPO_ROOT" && bash scripts/affected-tests.sh "$y" 2>/dev/null)"
Expand Down
209 changes: 209 additions & 0 deletions scripts/check-plugin-catalog-enablement.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
#!/usr/bin/env bash
# Gate: this repo's own catalog and its own enabled-plugin set must name the
# same plugins.
#
# scripts/check-plugin-catalog-enablement.sh run the gate (no flags)
#
# WHY. docs/CLOUD-SESSIONS.md states the property this repo depends on:
# "`enabledPlugins` turns on the whole catalog, so this repo dogfoods
# everything it publishes and a regression in any plugin surfaces here first."
# Nothing enforced it. Three plugins reached main with a catalog entry and no
# `enabledPlugins` key -- ai-slop (#2892), context-budget (#2932) and
# improvement (#2985) -- while the plugin PRs on either side of them
# (coupling #2913, overengineering #2961) remembered the settings entry. The
# failure is silent by construction: a plugin absent from `enabledPlugins` is
# simply never installed by .claude/cloud-bootstrap.sh, whose wanted-set is
# computed from that same file, so the session comes up green with the
# plugin's skills missing and no line of output naming what is absent. That is
# the docs/conventions/liveness-assertion/ shape -- a documented guarantee with
# no gate behind it -- and it costs exactly the dogfooding the directory-source
# marketplace exists to provide.
#
# NOT COVERED ELSEWHERE. plugins/claude-config/skills/audit/scripts/
# check-plugin-drift.sh audits this same axis for CONSUMER repos, but it
# resolves each marketplace through `source.repo` and records SKIP for a
# marketplace that declares none. This repo's marketplace is a relative
# `directory` source with no `repo` field, so that detector structurally
# cannot see this repo's own drift. scripts/check-plugin-manifest-presence.sh
# holds the catalog against the filesystem (manifest present, name matches,
# no unregistered directory); it says nothing about whether a catalogued
# plugin is ever enabled.
#
# WHAT IS CHECKED (both directions, so the two sets are provably equal):
# 1. UNENABLED PLUGIN -- a .claude-plugin/marketplace.json entry with no
# `<name>@<marketplace>` key in .claude/settings.json `enabledPlugins`.
# The class that shipped three times.
# 2. ORPHANED ENTRY -- an `enabledPlugins` key for this marketplace that
# names no catalog entry. What a plugin rename or removal leaves behind;
# the id resolves to nothing and the install silently no-ops.
# 3. UNSORTED KEYS -- `enabledPlugins` keys out of byte order. The same
# doc calls the alphabetical one-per-line layout the reason a single
# plugin can be flipped to `false` without disturbing the rest, and an
# insertion at the wrong point is how a duplicate-looking near-miss hides.
#
# A key set to `false` PASSES. An explicit `false` is a recorded decision --
# the documented off switch for the two entries whose MCP servers need
# credentials this environment has no reason to hold (`miro`, `dometrain`) --
# whereas an absent key is the drift this gate exists to name. The two are
# different states and only one of them is silent.
#
# Exit codes: 0 clean, 1 drift, 2 fatal (inputs missing or unreadable).
#
# Env overrides (tests only):
# PLUGIN_CATALOG_ENABLEMENT_MARKETPLACE -- path to marketplace.json
# PLUGIN_CATALOG_ENABLEMENT_SETTINGS -- path to settings.json
set -euo pipefail

cd "$(dirname "${BASH_SOURCE[0]}")/.."

if ! command -v jq >/dev/null 2>&1; then
echo "check-plugin-catalog-enablement: jq is required but not installed" >&2
exit 2
fi

MARKETPLACE="${PLUGIN_CATALOG_ENABLEMENT_MARKETPLACE:-.claude-plugin/marketplace.json}"
SETTINGS="${PLUGIN_CATALOG_ENABLEMENT_SETTINGS:-.claude/settings.json}"
BOOTSTRAP="${PLUGIN_CATALOG_ENABLEMENT_BOOTSTRAP:-.claude/cloud-bootstrap.sh}"

for f in "$MARKETPLACE" "$SETTINGS"; do
if [[ ! -f "$f" ]]; then
echo "check-plugin-catalog-enablement: $f not found" >&2
exit 2
fi
if ! jq empty "$f" 2>/dev/null; then
echo "check-plugin-catalog-enablement: $f is not valid JSON" >&2
exit 2
fi
done

if [[ ! -f "$BOOTSTRAP" ]]; then
echo "check-plugin-catalog-enablement: $BOOTSTRAP not found" >&2
echo " Without it the marketplace-identity coherence check below cannot run, and a green" >&2
echo " result would overstate what this gate verified. Failing rather than skipping." >&2
exit 2
fi

# The marketplace this repo declares for itself. Derived rather than
# hardcoded, so renaming the marketplace does not leave the gate silently
# checking a suffix nothing uses any more. Exactly one is expected: a second
# would make "the catalog" ambiguous, and guessing which one to hold the
# catalog against is how a gate goes quietly wrong.
mapfile -t market_keys < <(jq -r '.extraKnownMarketplaces // {} | keys[]' "$SETTINGS" | tr -d '\r')
Comment thread
kyle-sexton marked this conversation as resolved.

if ((${#market_keys[@]} != 1)); then
printf 'check-plugin-catalog-enablement: expected exactly one entry in %s extraKnownMarketplaces, found %d.\n' \
"$SETTINGS" "${#market_keys[@]}" >&2
printf ' This gate holds %s against the plugins this repo enables for itself;\n' "$MARKETPLACE" >&2
printf ' with %d marketplaces declared there is no single set to hold it against.\n' "${#market_keys[@]}" >&2
exit 2
fi

MARKET="${market_keys[0]}"

# CRLF tolerance: jq on Windows (Git Bash) can emit \r\n, and a trailing \r
# would ride into every comparison below as an invisible suffix mismatch.
catalog="$(jq -r '.plugins[]?.name // empty' "$MARKETPLACE" | tr -d '\r' | sort -u)"

if [[ -z "$catalog" ]]; then
echo "check-plugin-catalog-enablement: $MARKETPLACE lists no plugins" >&2
exit 2
fi

# Keys in declaration order (for the sort check) and the plugin names they
# carry for this marketplace (for the set comparisons).
declared_keys="$(jq -r '.enabledPlugins // {} | keys_unsorted[]' "$SETTINGS" | tr -d '\r')"

# Suffix stripping is a LITERAL parameter expansion, never a `sed` expression
# interpolating $MARKET. A marketplace name is free-form text from a settings
# file, so a regex metacharacter in it would change what the pattern matches
# rather than what it says: a name like 'melodic.software' would make the '.'
# match any character, silently accepting 'alpha@melodicXsoftware' as this
# marketplace's key and reporting the real plugin as unenabled. A gate whose
# whole purpose is to catch silent drift must not have a silent-wrongness path
# of its own. Flagged as informational (not a finding) by the security review
# on #3235, on the grounds that the value is repo-controlled and crosses no
# trust boundary -- true, and beside the point for correctness.
enabled="$(
while IFS= read -r key; do
[[ -n "$key" ]] || continue
[[ "$key" == *"@$MARKET" ]] || continue
printf '%s\n' "${key%"@$MARKET"}"
done <<<"$declared_keys" | sort -u
)"

errors=0

# 1. FORWARD -- catalogued but never enabled.
while IFS= read -r name; do
[[ -n "$name" ]] || continue
printf "UNENABLED PLUGIN: %s catalogs '%s', but %s enabledPlugins has no '%s@%s' key.\n" \
"$MARKETPLACE" "$name" "$SETTINGS" "$name" "$MARKET" >&2
printf " .claude/cloud-bootstrap.sh computes what it installs from that file, so '%s' never loads in a session here.\n" \
"$name" >&2
errors=$((errors + 1))
done < <(comm -23 <(printf '%s\n' "$catalog") <(printf '%s\n' "$enabled"))

# 2. INVERSE -- enabled but no longer catalogued.
while IFS= read -r name; do
[[ -n "$name" ]] || continue
printf "ORPHANED ENABLED ENTRY: %s enables '%s@%s', but %s catalogs no such plugin.\n" \
"$SETTINGS" "$name" "$MARKET" "$MARKETPLACE" >&2
printf " The id resolves to nothing; the install silently no-ops. Drop the entry, or restore the catalog entry it names.\n" >&2
errors=$((errors + 1))
done < <(comm -13 <(printf '%s\n' "$catalog") <(printf '%s\n' "$enabled"))

# 3. LAYOUT -- keys must stay in byte order, one per line.
if [[ -n "$declared_keys" ]]; then
if ! diff <(printf '%s\n' "$declared_keys") <(printf '%s\n' "$declared_keys" | LC_ALL=C sort) >/dev/null; then
printf 'UNSORTED enabledPlugins: keys in %s are not in byte order.\n' "$SETTINGS" >&2
printf ' docs/CLOUD-SESSIONS.md documents the alphabetical one-per-line layout as what lets a single\n' >&2
printf ' plugin be flipped to false without disturbing the rest. First keys out of order:\n' >&2
diff <(printf '%s\n' "$declared_keys") <(printf '%s\n' "$declared_keys" | LC_ALL=C sort) |
sed -n '1,10p' | sed 's/^/ /' >&2
errors=$((errors + 1))
fi
fi

# 4. IDENTITY -- the bootstrap must agree on which marketplace this is.
# This gate derives the suffix from extraKnownMarketplaces; .claude/cloud-
# bootstrap.sh hardcodes `marketplace_name` and selects its install set with
# endswith("@" + $n). Rename the marketplace and update both settings keys and
# the catalog, and the two disagree: `wanted` matches nothing, cloud sessions
# install none of the catalog, and this lane stays green over it -- a parity
# gate certifying a set nobody installs. Raised as P2 by the Codex review on
# #3235. Verifying the constant rather than deriving it keeps the bootstrap's
# behavior byte-identical (it must work before anything else does) while making
# a rename that touches only one of the two impossible to merge.
bootstrap_name="$(sed -n 's/^marketplace_name="\([^"]*\)".*/\1/p' "$BOOTSTRAP" | head -1)"

if [[ -z "$bootstrap_name" ]]; then
printf 'UNREADABLE BOOTSTRAP IDENTITY: %s declares no marketplace_name="..." assignment at line start.\n' \
"$BOOTSTRAP" >&2
printf ' This gate cannot confirm the bootstrap installs the marketplace it holds the catalog against.\n' >&2
printf ' If the constant moved or was renamed, update this check with it.\n' >&2
errors=$((errors + 1))
elif [[ "$bootstrap_name" != "$MARKET" ]]; then
printf "MARKETPLACE IDENTITY MISMATCH: %s declares marketplace_name='%s', but %s declares the marketplace '%s'.\n" \
"$BOOTSTRAP" "$bootstrap_name" "$SETTINGS" "$MARKET" >&2
printf " The bootstrap selects what it installs with endswith(\"@%s\"), so it would install none of the\n" \
"$bootstrap_name" >&2
printf " '%s' catalog this gate just checked. Update both, or the parity below certifies a set nobody installs.\n" \
"$MARKET" >&2
errors=$((errors + 1))
fi

if ((errors > 0)); then
{
echo
echo "Every $MARKETPLACE entry must carry an enabledPlugins key in"
echo "$SETTINGS, and every enabledPlugins key for the '$MARKET'"
echo "marketplace must name a catalogued plugin. A key set to false is a"
echo "recorded decision and passes; an absent key is drift and does not."
echo "$BOOTSTRAP must name that same marketplace, or what it installs and"
echo "what this gate checks are two different sets."
} >&2
exit 1
fi

catalog_count="$(printf '%s\n' "$catalog" | grep -c . || true)"
echo "Every one of the $catalog_count catalogued plugins carries an enabledPlugins key for '$MARKET'; none orphaned; keys sorted; $BOOTSTRAP installs that same marketplace."
Loading