Skip to content

test(source-control): add a setup eval case that writes only what the user confirmed - #8162

Merged
kyle-sexton merged 3 commits into
mainfrom
work-resolve-issue/issue-7998-39106008892-1
Oct 19, 2026
Merged

kyle-sexton merged 3 commits into
mainfrom
work-resolve-issue/issue-7998-39106008892-1

Conversation

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Closes #7998

Summary

Opened by the work-resolve-issue lane. The run's own summary:

Adds the plugin-eval case setup-writes-only-confirmed-change. It runs /source-control:setup apply layer=team in a scratch git repo, confirms the team convention file, and declines the personal overlay and the neutral convention file. Deterministic file_exists graders fail if either declined file appears.

Not verified

  • The eval was not run (paid runs are out of scope), so the graders have not been exercised against a live model.
  • check-evals-quality could not validate the case: it rejected the prompt.md path with a jq parse error. Not run successfully.
  • scaffold.sh executable bit: chmod was denied in this run, so the committed mode may be 644. Sibling scaffolds are 100755.

Fix

  • plugins/source-control/evals/setup-writes-only-confirmed-change/case.yaml: case definition using scaffold.sh
  • plugins/source-control/evals/setup-writes-only-confirmed-change/scaffold.sh: scratch repo with one commit and the .claude/**/.local. ignore rule already present
  • plugins/source-control/evals/setup-writes-only-confirmed-change/prompt.md: setup apply prompt that confirms one write and declines two
  • plugins/source-control/evals/setup-writes-only-confirmed-change/graders/: skill-fired, team-file-written, no-overlay-written, no-neutral-file-written
  • .changes/source-control/test-setup-eval-writes-only-confirmed-8e4b2c61.md: patch changelog fragment

Verification

The run reports:

Left out: the items under Not verified.

Related

#7998

… user confirmed

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed ad660ea7e3252b96e182ec1dd3f251b00f59781f: 0 finding(s), 0 as inline comments.

Security review (incremental, head ad660ea7)

No security issues found.

The only file changed since e5c433b0 is plugins/source-control/evals/setup-writes-only-confirmed-change/scaffold.sh. The diff file held no patch, so I read the script at the head commit.

  • It runs git init in the eval workspace, sets a placeholder identity (eval@example.invalid), writes two fixed files and makes one commit.
  • It takes no external input, environment variables or arguments, so there is no injection or path-handling surface.
  • It handles no tokens, credentials or network calls, and nothing in it is machine-specific.
  • It uses set -euo pipefail.
  • The change doesn't delete, narrow or soften any standing instruction, so the instruction-surface check doesn't apply.

I did not re-review the rest of the PR, which was reviewed at e5c433b0.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed ad660ea7e3252b96e182ec1dd3f251b00f59781f: 0 finding(s), 0 as inline comments.

Incremental review of plugins/source-control/evals/setup-writes-only-confirmed-change/scaffold.sh since e5c433b. The incremental diff has no content patch, only a mode change (commit ad660ea, "mark shebang scripts executable"). I confirmed the file is now mode 755 in the checkout and read its contents.

The script content is correct and unchanged. It runs set -euo pipefail and creates a git repo on main with local identity set. It writes the .claude/**/*.local.* ignore rule and a README, then makes one commit. That matches the case's stated setup, a repo with no source-control config at any layer.

This resolves the "Not verified" note in the PR body that the scaffold might be committed as 644.

No findings. I did not re-review the other files, which are unchanged since the last review. I did not run the eval, so the graders have not been exercised against a live model.

…eval

setup sets disable-model-invocation, so the Skill tool cannot invoke it.
team-file-written already shows the skill ran.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 1 of 3 on e5c433b.

@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 18, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The pr-unblock lane stopped on this pull request and needs a person: a review finding is uncertain or deferred, so a person decides it.

  • Reason: feedback-uncertain
  • Resume key: e5c433b0b74158c1cf58e9feca5495fcaae3ede5 (the head the lane stopped on)

To resume, deal with the reason above, remove the needs-human label (and do-not-merge, if set), then dispatch the pr-unblock workflow with pr-number=8162 and activity=address-feedback:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=8162 -f activity=address-feedback

lint-repo's exec-bit gate refused these scripts at mode 100644.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit 80ae9d4 Oct 19, 2026
25 checks passed
@kyle-sexton
kyle-sexton deleted the work-resolve-issue/issue-7998-39106008892-1 branch October 19, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(source-control): add a setup eval case that writes only what the user confirmed

1 participant