[kyle-sexton]

Closes #9936

Summary

Commits the hand-rolled developer-experience eval scenario sources to
plugins/developer-experience/evals/sources/, so the porting children of #6649 can read them. The
sources come from the harness of PR #6668, which until now existed only in one host's gitignored
workspace.

Included (520 files):

• eval-scenarios-draft.md: the spec (S1-S7, T1-T4, N1, stays-quiet, rubrics, run matrix).
• harness/: scenarios.sh, run-cell.sh, run-batch.sh, the fixture generators
  (build-fixtures.py, build-agentic-overlay.py), the scorers and report helpers.
• held-out/: the H-audit and H-build scenarios.
• judge/: items.json (the J-item rubric checklist), judge.py, build_inputs.py, score_j.py,
  briefs/variant-brief.md.
• labels/: 484 labeled judge calibration samples (452 samples, 32 labels.jsonl).
• lists/: the run lists run-batch.sh reads.

Left out: the 200 generated fixture files (the generator reproduces them), every results*/ tree,
RUNS.md, and judge run output (judge/inputs/, judge/calibration/, judge/scores/,
judge/status.json). A .gitignore in sources/ keeps regenerated fixtures and new run output
out of the tree.

Fix

• Path math: the move adds one directory level, so build-fixtures.py and run-cell.sh now
  resolve the repository root four levels up.
• Machine paths: the hard-coded session scratch directories became required environment variables
  that fail when unset (EVAL_RUNS_DIR, EVAL_TEST_DIR, DX_JUDGE_SCRATCH). The judge input
  scrub now strips $DX_JUDGE_SCRATCH/sandboxes/<box>/ instead of a fixed path.
• harness/score-deterministic.test.sh is renamed score-deterministic-check.sh: it needs a real
  run's output, which is not committed, so the plugin test runner must not collect it as a suite.
• Labeled samples are judge inputs, so they stay byte-for-byte except for the path scrub below.
  Their typos and markdownlint exemptions landed first, in ci: exempt the developer-experience labeled eval samples from typos and markdownlint #10723.
• Seven scripts that start with #! are marked executable, as the exec-bit check requires.
• The developer-experience changelog fragment is bump: none: no shipped skill, script or setting
  changes.

Identifying-details review

The labeled samples are model output answering the synthetic scenario prompts on the fictional
tarnquill and brackenfold fixtures, not session or product transcripts, so the grader-sample
case of .claude/rules/eval-case-transcripts.md applies once no machine-specific path or identifying
detail remains. The review covered the whole committed set, plus the 11 files the issue counted:

File	Hit	Change
harness/run-cell.sh	session scratch path with a home-directory user name	EVAL_RUNS_DIR required
harness/score-deterministic.test.sh	same	EVAL_TEST_DIR required; file renamed
judge/judge.py	same	DX_JUDGE_SCRATCH required
judge/build_inputs.py	same, in the scratch constant and the scrub regex	DX_JUDGE_SCRATCH required; scrub derives from it
judge/briefs/variant-brief.md	worktree root under a home directory	repository-relative root
labels/T4/2/samples/{real,var}-p6b-A2-run2.md, labels/T4/3/samples/{real,var}-p6b-A2-run2.md	session scratch path inside a quoted command	<plugin-root>/ placeholder
harness/build-fixtures.py	five email addresses	kept: fictional names on reserved example.org, example.net and example.com domains
RUNS.md	home path	not committed (run log)

Other checks on the committed set:

• A case-insensitive search for user names, host names, /home/ and /Users/ paths, the session
  UID marker, and email addresses found nothing beyond the rows above. The remaining addresses are
  @example.invalid and the @AGENTS.md import.
• /tmp/accounts.jsonl in labels/S4/9b is fixture task content, not a host path.
• gitleaks dir on the committed set reported no leaks.

Verification

• Ran python3 -I harness/build-fixtures.py, then harness/build-agentic-overlay.py, from the
  committed location. diff -r against the original 200 fixtures differs only in the
  project GUIDs that dotnet new sln writes into the four Tarnquill.sln files. The output was
  deleted afterwards.
• scripts/run-ruff.sh check passes. shellcheck and shfmt -d are clean on harness/*.sh.
  markdownlint-cli2 is clean on the sources. typos is clean outside labels/.
• scripts/check-changelog-fragments.sh --check-required origin/main exits 0.
• scripts/check-affected.sh --tests on the merged head: 15 passed, 1 skipped (no changed skill file).
• A parallel run of the pre-push reference verifiers over the 464 new files found 28 dotnet build --nologo notes, all in judge samples. Since fix(guardrails): reference-verify skips recorded eval samples #10698, the hook skips evals/**/samples/.
• No eval or claude run was needed.

Related

• Parent: feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
• Source harness: PR feat(developer-experience): add developer-experience plugin with setup, build-cli and audit-tools #6668
• Lint exemptions: ci: exempt the developer-experience labeled eval samples from typos and markdownlint #10723. Reference-verify sample skip: fix(guardrails): reference-verify skips recorded eval samples #10698.

🤖 Generated with Claude Code

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
39229828	Triggered	Generic CLI Secret	9400cb8	plugins/developer-experience/evals/sources/harness/score-deterministic.py	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-28T22:02:22.883041Z	7bc2c87	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7bc2c87374

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

claude-review reviewed f350ec7beeb539d824ea3dafb4172eeec77c7d78: 0 finding(s), 0 as inline comments.

Incremental review of plugins/developer-experience/evals/sources/harness/run-cell.sh since 8e42649: no findings.

The change adds a guard at lines 100-101 that rejects a relative EVAL_RUNS_DIR with die 2. I read the surrounding script to check it. Exit 2 matches the script's other usage errors, and the guard runs before RUN_DIR is built. The worktree prefix check (line 106) and the parent walk for instruction files (lines 114-121) both need an absolute path, so the guard fits them. I ran nothing, so the behavior is checked by reading only.

The skip gate did not apply, because the file is a bash script under the evals tree and not a trivial or automated change. This lane's own criteria had nothing else to flag. I read the PR context file only for the closing-issue check and found no criterion this change affects. No security-lane scope, manifest, or cost items are hit.

[github-actions]

claude-security-review reviewed f350ec7beeb539d824ea3dafb4172eeec77c7d78: 0 finding(s), 0 as inline comments.

Incremental security review of plugins/developer-experience/evals/sources/harness/run-cell.sh (change since 8e42649). No security issues found.

The change adds one guard, [[ "$RUNS_ROOT" == /* ]] || die 2 ..., after EVAL_RUNS_DIR is read. It tightens a check and widens nothing.

• Guard behavior: die exists at line 44, and the guard runs before RUN_DIR is built and before the rm -rf on the replace path. A relative EVAL_RUNS_DIR now exits 2. Before, the in-worktree check at line 106 ("$RUN_DIR" == "$WORKTREE"/*) could not match a relative path, and the parent walk that looks for stray CLAUDE.md/AGENTS.md files never reached /. This closes that gap.
• Not covered: an absolute path containing .. is not normalized. That was true before this change, and the only input is the operator's own environment variable, so I rate it no finding.
• Surface: the change touches no trusted path, plugin manifest, token scope or merge hold. No instruction-surface deletion.

critical-findings: 0

[github-actions]

Post-merge mutation testing

• Range: f9f10cf..4ad980b
• Merged pull request: #10754
• Outcome: reported

Killed 2, survived 0, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 100%; mutation score 100%.

File	Killed	Survived	Timeout	No coverage	Invalid	Not run
scripts/check-line-citations.py	2	0	0	0	0	0

The run's summary has the full report.