ci: decide whether the lane may add the eval: run label to its own PRs #10023

Activity

1. github-actions

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
2. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   Contributor

   More actions

   Proposed work class: none

   Size: Small
   Complexity: Medium
   Uncertainty: Knowable
   Flags: intent, security
   Blocked by: none

   I understand this as a request for an owner decision on whether the work-resolve-issue lane may add eval: run to its own PRs. The auto-label was removed from #9835 as a trust-boundary change. Option (b) would let agent-written code run with the maintenance credential, and that is a security risk.

   Open question:

   1. Which option should apply: (a) no auto-label, (b) auto-label every lane PR, or (c) auto-label only when the change is confined to evals/ and descriptive skill text?
      RECOMMENDED: (a). Basis: feat(ci): opt-in PR label trigger for the changed plugin's eval suite #9705 scopes the eval as opt-in by label, and the owner kept the auto-label removed on 2026-10-25. (c) is the fallback if the owner wants automation, but it needs a reliable path check.

   Related: #9705, #9835.

   _{Automated triage. A maintainer reviews it.}
3. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   status: needs-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

   on Oct 25, 2026
4. github-actions

   added the

   Task

   issue type on Oct 25, 2026
5. melodic-automation-lanes commented on Oct 25, 2026

   

   melodic-automation-lanesBot

   on Oct 25, 2026

   Contributor

   More actions

   The triage verdict left this issue at status: needs-decision:

   • flagged: intent, security
   • held by status: needs-decision
   • no lanes App verdict comment shows whether an earlier verdict was flagged
6. melodic-automation-lanes

   removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 25, 2026
7. melodic-automation-lanes commented on Oct 25, 2026

   

   melodic-automation-lanesBot

   on Oct 25, 2026

   Contributor

   More actions

   Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

   Intended outcome: Option (a): the lane does not add eval: run; a listed human adds it, and the issue closes as decided.

   Related work

   • test(developer-experience): run the plugin-eval suite on the eval dispatch lane and record the delta #10005 (related): runs the plugin-eval suite on the eval dispatch lane; depends on how the label gets applied
   • feat(developer-experience): claude plugin eval suite in GitHub Actions #6649 (related): plugin eval suite in GitHub Actions, the workflow the label gates
   • chore(pr-pipeline): decide in the lanes audit whether lane run jobs keep sudo and docker #7791 (related): lane run job privileges audit, bears on running lane-written code with credentials

   Open questions

   1. Which option applies: (a) no auto-label, (b) auto-label every lane PR, or (c) auto-label only when the change is confined to evals/ and descriptive skill text?
      RECOMMENDED: (a) Do not auto-label; a listed human adds eval: run. Basis: judgment: the issue body says #​9705 scopes the eval as opt-in by label and the owner kept the auto-label removed on 2026-10-25; (b) lets agent-written code run with the maintenance credential with no person opting in.
   2. If the owner wants automation, may (c) be used, and what check defines 'descriptive skill text' reliably enough for an unattended run?
      RECOMMENDED: Only adopt (c) after a path allowlist (evals/ plus named skill-text files) is specified and verified against the full PR diff; otherwise stay with (a). Basis: judgment: skill text is read by the model, so a text-only change can still steer eval behavior; the path check must be exact.

   Sources read

   • /home/runner/work/_temp/refine-data/context.json: issue body asks for an owner decision among (a), (b), (c); no comment records a decision
   • repository grep for 'eval: run' in .base: no matches, so no current file mentions the label

   _{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}
8. kyle-sexton commented on Oct 26, 2026

   

   kyle-sexton

   on Oct 26, 2026

   MemberAuthor

   More actions

   Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

   go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

   I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

   No owner ledger (.work/*/interview-checklist.md, .work/gh-monitor-owner-decisions/) names this issue.

   1. Which option applies: (a) no auto-label, (b) auto-label every lane PR, or (c) auto-label only changes confined to evals/ and descriptive skill text?
      Decided: (a). The lane does not add eval: run; a listed human adds it. Basis: the body records the owner's earlier call on this exact change:

      keep it removed and propose it separately

      (b) and (c) would let lane-written code run with the maintenance credential with no person opting in, which the body names a trust-boundary change. This record widens no boundary, so it decides (a) and nothing more. If automation is wanted later, it comes back as its own proposal.

   Closed as done. Option (a) is the state on main: git grep "eval: run" at origin/main 68d3f2bd81 finds no file that names the label, so no code adds it and nothing is left to build. This record closes the issue.
9. kyle-sexton

   closed this as completedon Oct 26, 2026