Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions scripts/check-skill-portability.sh
Original file line number Diff line number Diff line change
Expand Up @@ -197,11 +197,101 @@ scan_file() {
# (branch.<name>.remote / @{upstream}). That is the false green the token
# file staged-class preamble forbids, so the scoping lands with the class
# set that made it reachable rather than with the class that trips over it.
#
# Stack/ecosystem class (#714, Stage 3) markers are CONSUMER-SEAM evidence:
# the ecosystem has to come from the consuming project, not from the skill.
# Two shapes qualify, both co-located on the hit line so a reviewer sees the
# evidence in the diff:
# 1. a userConfig read (`${user_config.<key>}`) — the consumer supplies
# the stack, which is the seam this class exists to force. Anchored to
# the read syntax, not the bare word: prose ABOUT a userConfig seam is
# not a seam. Scoped to the ecosystem LITERALS; the architecture token
# takes shape (2) alone, for the reason recorded at
# is_architecture_pattern below.
# 2. a neutral `<placeholder>` AND an explicit illustration lead (`e.g.`,
# `for example`, `such as`). The portable idiom already used across
# this corpus is a placeholder followed by a multi-ecosystem example
# run — `<build-command> # e.g. dotnet build / npm run build / cargo
# build` — where the ecosystem token illustrates the placeholder
# instead of being the value the skill ships.
# BOTH halves of (2) are required, and that is the whole guard. A
# placeholder alone would excuse `run dotnet build in <repo-root>`, which
# ships the stack and merely parameterizes the path; an illustration lead
# alone would excuse `e.g. run dotnet build`, the bare hardcode wearing a
# hedge. That generic hedge is exactly what the token file staged-class
# preamble forbids as a guard, so neither half is admitted on its own.
#
# Deliberately NOT a marker: "another ecosystem is named on the same line".
# A multi-ecosystem enumeration with no placeholder and no illustration lead
# (`.csproj`, `package.json`, `pyproject.toml`, and equivalents) is the
# dominant residual shape in the corpus and reads as correct, but a
# co-occurrence test cannot tell it from a line that hardcodes one stack and
# merely mentions another. It uses the per-site portability-ok escape until
# someone brings evidence for a tighter rule.
#
# Class membership is tested with index() on literal fragments, never with a
# regex match against the pattern text. The argument here is a PATTERN, so a
# regex test reads its metacharacters as syntax: `p ~ /[Dd]otnet/` asks for a
# D-or-d followed by "otnet" and therefore does NOT match the pattern STRING
# `[Dd]otnet`, whose `d` is followed by `]otnet`. That silently returns 0 for
# every member and the whole class goes unguarded while the tests that only
# check flagging still pass. The branch class above escapes its
# metacharacters for the same reason; index() removes the trap instead of
# re-solving it per pattern. Fragments are chosen to be metacharacter-free in
# the pattern text, so each one survives however its pattern is escaped.
#
# A fragment also has to be UNIQUE to its pattern, not merely present in it.
# `\.NET` is the literal as the staged pattern spells it; a bare `NET` is a
# substring of `NETRC`, `CONNECT`, and `INTERNET`, so a future token class
# carrying that trigram would silently inherit the consumer-seam guard of
# this class, and a real violation of it could be excused by an unrelated
# userConfig read. assert_staged pins the pattern spelling verbatim, so the
# fragment and the pattern it selects cannot drift apart.
function is_ecosystem_pattern(p) {
return index(p, "otnet") || index(p, "\\.NET") ||
index(p, "csproj") || index(p, "sln") || index(p, "razor") ||
index(p, "lazor") || index(p, "C#") || index(p, ".cs(")
}
# Architecture vocabulary is guarded by the illustration shape ALONE — never
# by a userConfig read. The token file entry for it names exactly two
# escapes, a `portability-scope` declaration and the
# placeholder-plus-illustration shape, and a userConfig read is neither: the
# class-wide marker reaching this token was an accident of class membership.
# It excused an architecture hardcode whenever ANY unrelated setting was read
# on the line — `Read ${user_config.test_command}, then place handlers per
# Clean Architecture` — a false green on the one token in this class that is
# ACTIVE. Evidence has to parameterize the coupling it excuses, and no
# userConfig KEY NAME can be trusted to prove that for an architecture.
function is_architecture_pattern(p) { return index(p, "Clean Arch") }
# Shared by both branches so the two cannot drift apart. The placeholder is
# lowercase-hyphenated by corpus convention, which includes single-letter CLI
# operands (`<n>`, `<s>`); uppercase `<SCREAMING-CASE>` is deliberately not
# accepted, because nothing in this corpus uses it and admitting it would
# widen the guard past every shape under test.
#
# KNOWN RESIDUAL, admitted deliberately: inline HTML shares this spelling
# space, so `See the <a> docs, e.g. run dotnet build` satisfies both halves
# with no real seam. One line of text cannot separate the two here — `<b>`,
# `<i>`, `<p>`, `<s>`, and `<n>` are all genuine CLI operands in this corpus,
# so a minimum length rejects real placeholders, and a tag denylist
# enumerates instances of an open-ended shape while leaving `<var>` or any
# invented token. This is the same limit the token file records for the
# multi-ecosystem enumeration, and it takes the same remedy — a per-site
# portability-ok — until someone brings evidence for a tighter rule.
function is_illustrated_placeholder(l) {
return l ~ /<[a-z][a-z0-9-]*>/ && l ~ /(e\.g\.|for example|such as)/
}
function is_guarded(l, p) {
if (p ~ /origin\/\(main\|master\)/) {
return l ~ /origin\/HEAD/ || l ~ /symbolic-ref/ || l ~ /merge-base/ ||
l ~ /baseRefName/ || l ~ /-> *origin\//
}
if (is_architecture_pattern(p)) {
return is_illustrated_placeholder(l)
}
if (is_ecosystem_pattern(p)) {
return l ~ /[$]\{user_config\./ || is_illustrated_placeholder(l)
}
return 0
}
# Pass 1: collect active ERE patterns from the token list.
Expand Down
275 changes: 275 additions & 0 deletions scripts/check-skill-portability.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -640,6 +640,281 @@ else
fi
rm -rf "$fx"

# =============================================================================
# Stage 3 (#714) — stack/ecosystem opinion class.
#
# The class ships one ACTIVE token (Clean-Architecture vocabulary, proven 0/0
# against the corpus) and a staged set of ecosystem literals. What is new here
# is the class-scoped guard: the ecosystem class is excused by CONSUMER-SEAM
# evidence, not by branch-resolution evidence, so these fixtures pin both that
# the seam is honored and that the seams do not cross between classes.
# =============================================================================

# --- the ecosystem class is flagged where nothing declares it --------------
ECO_TOKEN='[Dd]otnet'
et="$(tokenfile "$ECO_TOKEN")"

f="$(tmpfile 'Build the project with `dotnet build` before running the tests.')"
if out="$(scan_with "$et" "$f" 2>&1)"; then
fail "a bare 'dotnet build' should fail in an agnostic skill, got success: $out"
elif echo "$out" | grep -q "COUPLING: ${f}:1:"; then
ok "ecosystem class: a bare stack default fails with file:line"
else
fail "expected line 1 flagged for a bare stack default, got: $out"
fi
rm -f "$f"

# --- a userConfig seam declares the stack and is honored --------------------
# The acceptance criterion for this stage: the class is honored where a
# `userConfig` seam supplies the ecosystem, because that is the seam the class
# exists to force. The skill names the stack only as what it is NOT assuming.
f="$(tmpfile 'Run `${user_config.build_command}` — never assume dotnet build.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
ok "ecosystem class: a userConfig seam read on the line is honored"
else
fail "a '\${user_config.*}' seam read must excuse the ecosystem class"
fi
rm -f "$f"

# --- prose ABOUT a userConfig seam is not a seam ----------------------------
# Anchored to the read syntax: a sentence mentioning userConfig supplies nothing
# to the consumer and must not buy a bare hardcode past the gate.
f="$(tmpfile 'This skill has a userConfig seam, so run dotnet build here.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
fail "prose mentioning userConfig must NOT guard a bare stack default"
else
ok "ecosystem class: prose about a seam does not stand in for a seam read"
fi
rm -f "$f"

# --- placeholder PLUS illustration is the portable idiom and passes --------
f="$(tmpfile 'Run `<build-command>` — e.g. `dotnet build`, `npm run build`, `cargo build`.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
ok "ecosystem class: a placeholder with an explicit illustration lead is honored"
else
fail "a '<placeholder> ... e.g. <stack>' illustration must be honored"
fi
rm -f "$f"

# The guard accepts three illustration leads, so all three are exercised: a
# later edit dropping one from the regex has to fail here rather than pass a
# suite that only ever tried `e.g.`.
f="$(tmpfile 'Run `<build-command>` — for example `dotnet build` or `npm run build`.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
ok "ecosystem class: 'for example' is accepted as an illustration lead"
else
fail "'for example' must be accepted as an illustration lead"
fi
rm -f "$f"

f="$(tmpfile 'Run `<build-command>`, such as `dotnet build` or `cargo build`.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
ok "ecosystem class: 'such as' is accepted as an illustration lead"
else
fail "'such as' must be accepted as an illustration lead"
fi
rm -f "$f"

# --- each half of that guard alone is NOT enough ---------------------------
# A placeholder alone parameterizes the PATH while still shipping the stack.
f="$(tmpfile 'Run dotnet build from <repo-root> before continuing.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
fail "a placeholder alone must NOT guard a shipped stack default"
else
ok "ecosystem class: a placeholder without an illustration lead still flags"
fi
rm -f "$f"

# An illustration lead alone is the bare hardcode wearing a hedge.
f="$(tmpfile 'Build the project, e.g. run dotnet build at the repository root.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
fail "an 'e.g.' hedge alone must NOT guard a shipped stack default"
else
ok "ecosystem class: an illustration lead without a placeholder still flags"
fi
rm -f "$f"

# --- the per-site and whole-file escapes work for this class too -----------
f="$(tmpfile 'Run `dotnet build`. <!-- portability-ok: this plugin ships a .NET analyzer -->')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
ok "ecosystem class: a per-site portability-ok escape is honored"
else
fail "the ecosystem class must honor a per-site portability-ok escape"
fi
rm -f "$f"

f="$(tmpfile '<!-- portability-scope: this skill targets .NET solutions only -->
Run `dotnet build` to restore and compile.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
ok "ecosystem class: a whole-file portability-scope declaration is honored"
else
fail "the ecosystem class must honor a whole-file portability-scope declaration"
fi
rm -f "$f"

# --- guard markers are CLASS-SCOPED, proven in both directions -------------
# This is the failure the token file staged-class preamble warns about: without
# scoping, the first additional class to activate inherits the branch class's
# evidence as a blanket excuse. Branch-resolution evidence proves which BRANCH
# was resolved and says nothing about which ECOSYSTEM the skill assumes.
f="$(tmpfile 'Resolve the base with `git symbolic-ref refs/remotes/origin/HEAD`, then run dotnet build.')"
if scan_with "$et" "$f" >/dev/null 2>&1; then
fail "branch-resolution evidence must NOT excuse an ecosystem hardcode"
else
ok "ecosystem class: branch-detection evidence does not excuse a stack default"
fi
rm -f "$f"

# And the converse: a consumer-seam read proves nothing about the branch.
f="$(tmpfile 'Read `${user_config.build_command}`, then diff against origin/main.')"
if scan_paths "$f" >/dev/null 2>&1; then
fail "a userConfig seam read must NOT excuse a bare branch default"
else
ok "branch class: an ecosystem seam read does not excuse a branch default"
fi
rm -f "$f"

rm -f "$et"

# --- every staged ecosystem literal is COVERED by the class-membership test -
# assert_staged below pins the pattern TEXT; this pins that is_ecosystem_pattern
# still recognizes that text. The two are separate failures: a ninth literal
# staged with no matching fragment in the function flags correctly and is
# silently UNGUARDED, so activating it would reject the very consumer seam the
# class exists to force — a false red that assert_staged cannot see.
#
# Each literal is asserted twice, because the guarded case alone is vacuous: a
# fixture that never matched its pattern also produces a clean scan. The control
# proves the hit exists, then the guarded line proves the fragment covers it.
# Flat pattern/sample pairs rather than delimited records: every ERE here is
# free to contain any delimiter a split would need, `|` included.
STAGED_COVERAGE=(
'[Dd]otnet' 'dotnet build'
'(^|[^a-zA-Z0-9_])\.NET([^a-zA-Z0-9_]|$)' 'the .NET SDK'
'\.csproj' 'src/App.csproj'
'(^|[^a-zA-Z0-9_])C#([^a-zA-Z0-9_]|$)' 'the C# compiler'
'[Bb]lazor' 'Blazor components'
'\.sln([^a-zA-Z0-9_]|$)' 'App.sln here'
'\.razor' 'Page.razor here'
'\*\*/\*\.cs([^a-zA-Z0-9_]|$)' 'the **/*.cs glob'
)
for ((i = 0; i < ${#STAGED_COVERAGE[@]}; i += 2)); do
pat="${STAGED_COVERAGE[i]}"
sample="${STAGED_COVERAGE[i + 1]}"
tf="$(tokenfile "$pat")"
control="$(tmpfile "Then run ${sample} at the repository root.")"
guarded="$(tmpfile "Read \`\${user_config.build_command}\` — never assume ${sample}.")"
if scan_with "$tf" "$control" >/dev/null 2>&1; then
fail "coverage fixture for '$pat' does not match it — the assertion would be vacuous"
elif scan_with "$tf" "$guarded" >/dev/null 2>&1; then
ok "ecosystem class membership: '$pat' is recognized by the guard"
else
fail "staged ecosystem literal '$pat' is not covered by is_ecosystem_pattern"
fi
rm -f "$tf" "$control" "$guarded"
done

# --- the Clean-Architecture token is ACTIVE in the shipped list ------------
# Activated at 0 hits / 0 files against the corpus, so this asserts the shipped
# list enforces it rather than leaving it staged like the ecosystem literals.
f="$(tmpfile 'Place the handler in the Application layer per Clean Architecture.')"
if out="$(SKILL_PORTABILITY_TOKENS="$REAL_TOKENS" bash "$SCRIPT" --paths "$f" 2>&1)"; then
fail "Clean Architecture vocabulary should be enforced by the shipped list: $out"
elif echo "$out" | grep -q "COUPLING: ${f}:1:"; then
ok "Clean-Architecture vocabulary is ACTIVE in the shipped token list"
else
fail "expected line 1 flagged for Clean-Architecture vocabulary, got: $out"
fi
rm -f "$f"

# The same phrase under a declared scope is the intended escape, so the active
# token cannot strand a skill whose subject genuinely is that architecture.
f="$(tmpfile '<!-- portability-scope: this skill teaches Clean Architecture itself -->
Place the handler in the Application layer per Clean Architecture.')"
if SKILL_PORTABILITY_TOKENS="$REAL_TOKENS" bash "$SCRIPT" --paths "$f" >/dev/null 2>&1; then
ok "Clean-Architecture vocabulary is excused by a declared narrower scope"
else
fail "an active Clean-Architecture token must honor portability-scope"
fi
rm -f "$f"

# The illustration shape is the other escape the token file names, so a skill
# presenting one architecture among several is not stranded either.
f="$(tmpfile 'Organize by `<architecture-style>` — e.g. Clean Architecture, vertical slices.')"
if SKILL_PORTABILITY_TOKENS="$REAL_TOKENS" bash "$SCRIPT" --paths "$f" >/dev/null 2>&1; then
ok "Clean-Architecture vocabulary is excused by placeholder-plus-illustration"
else
fail "the architecture token must honor the placeholder-plus-illustration shape"
fi
rm -f "$f"

# An UNRELATED userConfig read is not evidence about the architecture. The
# ecosystem literals take a `${user_config.*}` read as their consumer seam; the
# architecture token must not inherit it, or any skill reading any setting on the
# line buys a hardcoded architecture past the gate.
f="$(tmpfile 'Read `${user_config.test_command}`, then place handlers per Clean Architecture.')"
if SKILL_PORTABILITY_TOKENS="$REAL_TOKENS" bash "$SCRIPT" --paths "$f" >/dev/null 2>&1; then
fail "an unrelated userConfig read must NOT excuse a hardcoded architecture"
else
ok "architecture token: an unrelated userConfig read does not excuse it"
fi
rm -f "$f"

# --- the architecture token is PHRASE-DELIMITED ----------------------------
# The boundaries are load-bearing, not decoration: without them the token begins
# inside a preceding word and its optional suffix stops short, so ordinary prose
# is reported as a portability violation.
f="$(tmpfile 'Move the tarball to the Clean Archive directory before pruning.')"
if SKILL_PORTABILITY_TOKENS="$REAL_TOKENS" bash "$SCRIPT" --paths "$f" >/dev/null 2>&1; then
ok "architecture token: 'Clean Archive' is not a Clean-Architecture hit"
else
fail "'Clean Archive directory' must not flag — the optional suffix stopped short"
fi
rm -f "$f"

f="$(tmpfile 'The reviewer called the result UnClean Architecture, half-migrated.')"
if SKILL_PORTABILITY_TOKENS="$REAL_TOKENS" bash "$SCRIPT" --paths "$f" >/dev/null 2>&1; then
ok "architecture token: a match cannot begin inside a preceding word"
else
fail "'UnClean Architecture' must not flag — the token needs a left boundary"
fi
rm -f "$f"

# --- the ecosystem literals stay staged, verbatim --------------------------
# Same discipline as Stage 2: a staged pattern edited into a shape that no
# longer catches its own defect fails here rather than rotting unnoticed. Every
# spelling is POSIX-safe — a `\b` anywhere in this class is the inert-pattern
# trap the token file documents, so these literals also pin that the `\b`
# spellings did not come back.
for staged in \
'[Dd]otnet' \
'(^|[^a-zA-Z0-9_])\.NET([^a-zA-Z0-9_]|$)' \
'\.csproj' \
'(^|[^a-zA-Z0-9_])C#([^a-zA-Z0-9_]|$)' \
'[Bb]lazor' \
'\.sln([^a-zA-Z0-9_]|$)' \
'\.razor' \
'\*\*/\*\.cs([^a-zA-Z0-9_]|$)'; do
assert_staged 'ecosystem literal' "$staged"
done

# Scoped to ACTIVE lines: the token file necessarily QUOTES `\b` in the prose
# that documents why it is forbidden, and a commented line is indistinguishable
# from that prose by shape alone. The staged half is covered by the verbatim
# assert_staged fixtures above, which is the same discipline Stage 2 uses.
#
# The search itself is written as a bracket expression, `[\]b`, rather than the
# escape it is looking for: this file is scanned by the sibling
# check-shell-portability.sh gate, whose own token class forbids a
# backslash-b in a shell regex for exactly the POSIX reason under test. Spelling
# the needle as a bracketed literal keeps this assertion from tripping the gate
# that agrees with it.
if grep -vE '^[[:space:]]*(#|$)' "$REAL_TOKENS" | grep -q '[\]b'; then
fail "a backslash-b word-boundary escape reached an ACTIVE token — POSIX ERE does not define it"
else
ok "no ACTIVE token carries a backslash-b word-boundary escape"
fi

echo
echo "PASS=$PASS FAIL=$FAIL"
[[ "$FAIL" -eq 0 ]]
Loading