Add the interface engine - verify a workflow's contract by name (#374, PR-B) - #375
Conversation
… PR-B) Consume fidelity: interface. The audit now checks that a carried workflow honors its fixed contract without touching the owned body. - audit.py: split_jobs slices a workflow into job blocks by indent (no YAML parser); check_interface verifies required job keys, the ruleset-bound check name, the release-asset-<branch>-<target> handoff, and per-job required (pattern:/merge-multiple:) and forbidden (artifact-ids:) tokens. All findings are DRIFT - the body is owned and a rename is a hint to verify. An interface unit's absence is DRIFT too, not a hard letter, since a workflow's naming varies more than a carried config. - files.json: two workflow interface entries - test-pull-request.yml (universal: the check-workflow-status job and its ruleset-bound name) and build-release-task.yml (build types: the get-version/github-release jobs, the artifact handoff, and the no-artifact-ids fork rule). - audit.py --selftest exercises the engine on synthetic fixtures (7 cases: conformant, missing job+name, renamed check, artifact-ids fork, missing merge-multiple, an owned extra leaf job, and split_jobs). - AUDIT.md notes the new check. Also fix pre-existing clause-joining semicolons in this file's comments and docstrings so they do not fester. Verified: --selftest 7/7 pass, and the hub's own canonical test-pull-request.yml and build-release-task.yml are conformant to their contracts (0 findings). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds an interface-fidelity audit path so workflow baseline entries can be verified by their contracted wiring (job keys, required job name:, artifact handoff conventions, and forbidden download patterns) without requiring verbatim workflow bodies. This fits the repo’s broader “carry model” governance by making workflow conformance checkable in a structured, low-false-positive way.
Changes:
- Extend
spec/audit.pywith a lightweight workflow “jobs” slicer (split_jobs) and an interface-contract checker (check_interface), plus a--selftestfixture suite. - Add
fidelity: interfaceentries tospec/files.jsonfortest-pull-request.ymlandbuild-release-task.ymlwith targeted contract fields. - Update
AUDIT.mdto document the new interface-conformance check and its DRIFT-only semantics.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| spec/files.json | Adds two fidelity: interface baseline entries with workflow-specific contract fields. |
| spec/audit.py | Implements the interface engine (split_jobs, check_interface) and wires it into the audit with --selftest. |
| AUDIT.md | Documents the new workflow interface conformance check as part of the mechanized audit. |
Copilot round-1: "Three recursive trees calls" reads wrong; correct to "tree calls", consistent with "any tree is truncated" in the same sentence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot round-2 findings: - split_jobs: a dedented comment after the jobs children no longer scans into a job block or ends the mapping early - it is skipped, so a trailing or between-jobs top-level comment is not mis-attributed. Added a split_jobs test with a trailing top-level comment. - check_interface: skip the require/forbid token checks for a job that is absent - the requiredJobKeys check already reports the absence, so no redundant "job missing required token" for every token it cannot contain. Added a self-test: an absent required job reports exactly once. Verified: --selftest 8/8, hub canonical workflows still conformant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot round-3 findings: check_interface used a raw substring search over the whole workflow text and job blocks, so a token mentioned only in a comment counted as signal. A carried task file documents its own contract in comments (build-release-task.yml names release-asset- and artifact-ids: in prose), so this could false-pass a missing handoff and false-flag a forbidden token that appears only in a comment. Scan a code view (comment-only lines dropped) for requiredCheckName, artifactNameToken, and the per-job require/forbid tokens. Added self-tests: a forbidden token only in a comment is ignored, and a required token only in a comment does not count. Verified: --selftest 10/10, hub canonical workflows still conformant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
spec/audit.py:205
- check_interface() treats requiredCheckName as a raw substring match across the whole workflow (minus comment-only lines). That can false-pass if the string appears in some other field (including the workflow-level
name:) and does not actually appear as a jobname:in the required job. Since the contract is about the ruleset-bound job check name, match it as aname:line (and prefer searching within the required job block when there is exactly one required job).
name = contract.get("requiredCheckName")
if name and name not in code:
findings.append(("DRIFT", f"interface: {path} missing the ruleset-bound check name '{name}'"))
tok = contract.get("artifactNameToken")
if tok and tok not in code:
findings.append(("DRIFT", f"interface: {path} missing the '{tok}<branch>-<target>' artifact handoff"))
Copilot round-4: - split_jobs: _JOB_KEY now matches a key with an inline value, and the key line is included in its block, so an inline-mapping job (rare, but valid) is captured with its content. Added inline-mapping and trailing-comment split_jobs self-tests. - check_interface: match requiredCheckName as a `name:` field value (not a raw substring), so the ruleset-bound check name appearing in some other field (a run step, the workflow-level name) does not false-pass. Added a self-test for the check name present only in a run step. Verified: --selftest 13/13, hub canonical workflows still conformant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
spec/audit.py:205
- check_interface()'s requiredCheckName scan matches any
name:line anywhere in the workflow (including step-levelname:). That can false-pass a workflow where the ruleset-bound job name is missing but a step happens to use the same name. Since the contract is explicitly about the ruleset-bound job name, restrict the match to job-levelname:keys (the first-level keys inside each job block) and optionally add a selftest case where a stepname:equals requiredCheckName but the job-level name is absent.
name = contract.get("requiredCheckName")
if name and not re.search(r"^\s*name:\s*['\"]?" + re.escape(name) + r"['\"]?\s*$", code, re.M):
findings.append(("DRIFT", f"interface: {path} missing the ruleset-bound check name '{name}' as a job name"))
tok = contract.get("artifactNameToken")
if tok and tok not in code:
findings.append(("DRIFT", f"interface: {path} missing the '{tok}<branch>-<target>' artifact handoff"))
Copilot round-5 low-confidence finding (correct): the requiredCheckName scan
matched any name: line, so a step named the same as the ruleset-bound check
could false-pass while the job name was absent. Add job_level_names (a job's
own direct-child name key, at the shallowest block indent, excluding step names
and - name: list items) and match requiredCheckName against it. Added
self-tests for the check name appearing only as a step name or only in a run
step.
Verified: --selftest 14 cases pass, and the hub's test-pull-request.yml job
names ('Check pull request workflow status job', 'Validate sources job') are
extracted at job level so it stays conformant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second of the fidelity series (#374). Consumes
fidelity: interface: the audit now verifies a carried workflow honors its fixed contract by name and wiring, never the owned body — the virtual-function signature check.Change
spec/audit.pysplit_jobs(text)— slices a workflow into{job_key: block}by the jobs-level indent, no YAML parser (stdlib), mirroringheading_texts.check_interface(path, contract, text)— required job keys, the ruleset-bound checkname:, therelease-asset-<branch>-<target>handoff, and per-job required (pattern:/merge-multiple:) and forbidden (artifact-ids:) tokens. All findings DRIFT.interfaceunit's absence is DRIFT too (not a hard LETTER) — a workflow's naming varies more than a carried config, so everything here is advisory.--selftest(no network) — 7 fixture cases.spec/files.json— two workflowinterfaceentries:test-pull-request.yml(appliesTo: "*", the universalcheck-workflow-statusjob + its ruleset-bound name) andbuild-release-task.yml(build types, theget-version/github-releasejobs + artifact handoff + no-artifact-ids:rule).AUDIT.md— notes the new check, and fixes pre-existing clause-joining semicolons in this file's comments/docstrings so they don't fester.Verified
python3 spec/audit.py --selftest→ 7/7 (conformant, missing job+name, renamed check, artifact-ids fork, missing merge-multiple, owned extra-leaf-job no-finding, split_jobs).test-pull-request.ymlandbuild-release-task.ymlare conformant (0 findings).validate.pygreen (contracts pass PR-A's value-type guards), lints clean.Scoping of
build-release-task.ymlis best-effort (excludes source-only/docs to avoid false "absent" DRIFTs); any mis-scope only yields advisory DRIFTs to triage. It refines as downstreams are audited. Held ondevelop. PR-C adds the verbatim engine.