Repository navigation
fix(goal-draft): keep drafts concise and verifiable - #11284
Conversation
E2E report: goal-draft qualityTested on macOS, Node.js 24.14.1, default model qwen3.8-max. Baseline used global qwen 0.23.0; verification used the locally built CLI. Four fixed Chinese prompts ran in a small isolated workspace containing a documented failing addition test and two documented audit surfaces. Runtime data was isolated from the fixture and real user config was unchanged. Both versions were invoked with
All fixture file hashes remained unchanged. Recorded calls were limited to Goal reads, file reads, searches and tool discovery; no task tests, shell execution, file modifications or Goal creation occurred during drafting. The tighten case supplied text without an active Goal; live active-Goal edit/replace was not exercised. No terminal or Web Shell approval interaction was exercised. Limits: the concrete-test draft still claimed that matching an assertion with grep proves the test was not modified. This overclaim also existed in the baseline and remains a model-quality limitation. Three completed samples are not a guarantee that every generated criterion is sound. Different baseline/source versions, stochastic outputs and retry limits also prevent a latency-improvement claim. The forced deepseek provider probe failed with HTTP 401 before producing tokens; subsequent samples used the working default provider. Deterministic checks passed: full build/bundle/typecheck, bundled-asset equality, 69 focused tests, formatting and changed-test lint. Local automated review timed out after six minutes without a verdict; an independent read-only diff review completed with no actionable findings. 中文摘要:四类真实模型采样中三类得到最终输出,确认了默认停止假设、草稿尚未应用提示,以及缺失判据时不交付可执行命令。审计场景两次超时,零缺陷和覆盖配额的实际输出尚未验证。所有样本只读,未创建 Goal。完整构建、类型检查和 69 个测试通过;自动审查超时后,独立只读 diff 审查未发现可操作问题。仍保留模型过度解读 grep 证据的既有局限,不宣称所有草稿都可靠或生成速度提升。 |
|
@qwen-code /takeover |
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. This is a fork PR, so the first round comes from the next scheduled scan (usually within minutes). Remove the 中文说明🤝 已接管:autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突,直到移除标签或达到轮次上限。本 PR 来自 fork,首轮处理将由下一次定时扫描执行(通常几分钟内)。移除 |
|
🔀 Base updated: red check(s) [review-pr] pass on current main — merged current main via update-branch; CI will re-run. 中文说明🔀 已更新 base:红色检查 [review-pr] 在当前 main 上通过 —— 已通过 update-branch 合入当前 main,CI 将重新运行。 |
qwen-code-dev-bot
left a comment
There was a problem hiding this comment.
Approved at head a253680b.
The claim this change rests on is true at the registration level, which is what makes it worth more than a wording tweak. ProposeGoalTool is registered only when resolveInteractionMode(this) === 'interactive' (packages/core/src/config/config.ts:9440-9446), so a Web Shell or other ACP session has no dialog to promise — and the only places Web Shell mentions the tool at all are a display label and a translation string (toolFormatting.ts:26, i18n.tsx:3703). Telling the skill to print a copyable command there, and to stop saying "I'll propose it for approval", is therefore correcting a promise the runtime cannot keep rather than simplifying prose. The same goes for Budget: nothing parses the objective for a turn or minute count, the enforced ceiling is model.goalTokenBudget, and the 1,500-character cap the docs cite matches PROPOSE_GOAL_OBJECTIVE_MAX_CHARACTERS.
I checked the guard suite mechanically instead of trusting that it passes. All 53 string assertions in SKILL.test.ts hold against the head's SKILL.md; the three that do not appear in the body are the deliberate not.toContain assertions on allowedTools (run_shell_command, write_file, edit, update_goal), which is the right polarity for a skill that must draft without doing. The new ordering pin is the one I would have asked for: it locates the Needs clarification gate relative to ## Step 5 and Then hand off, and nothing else: (10047 / 10909 / 11214 at this head) and then slices the text between them to confirm the withholding instructions live inside that window — so moving the gate below the hand-off, which would let the skill print a runnable /goal set for an incomplete draft, goes red. A missing anchor yields -1 and fails toBeGreaterThan, so the pin cannot pass by absence.
One free improvement this PR was the natural place for. The same page, at docs/users/features/goals.md:19, still says "the footer pill and every status card show what it has spent" and "Once a Goal has billed a turn". Neither holds: the legacy goal_status card renders no figure in both renderers, and the four implementations gate on recorded spend rather than turns. That text arrived from PR #11248 and is already on main, so it is not this diff's defect and I am not blocking on it — but since this PR is editing that file and that page is where a reader learns what the budget line means, scoping the sentence here would have cost one line. Worth folding into the follow-up on the other PR rather than leaving for a third pass.
CI facts. At this head Test (ubuntu-latest), Lint & Static and Integration Tests (no-AK) were still queued behind the merge of main that landed a few minutes earlier; Classify PR, both Desktop Shell lanes and the precheck/assign/label jobs are green, and nothing has reported red. There are no review threads on the PR, and the automated review on this head is already an approve — my decision is on the content, not waiting on the board.
4a2c036
|
🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈(第 1/100 轮)。改动内容与我反驳保留之处如下: Address-review round — PR #11284Commit: All seven inline findings were Suggestion-level, valuable, and inside this PR's own footprint ( Feedback dispositions
No finding was declined, deferred, or escalated; Notes: after the exemplar row grew, Prettier re-padded the Weak→strong table (whitespace-only; row wordings unchanged). No settings source changed, so no schema regeneration. Mutation probesEach probe mutated SKILL.md, re-ran the focused suite, and restored the file:
Verification
Integration tests were not run: the change is skill prose plus its contract tests, fully exercised by the focused Vitest suite; it is not behavior reachable only through the bundled CLI or integration harness. 中文说明评审处理轮次 — PR #11284提交: 七条行内发现均为 Suggestion 级别、有价值、且位于本 PR 自身足迹内( 各条发现的处理
没有发现被拒绝、推迟或升级;因此省略 说明:示例行变长后,Prettier 重新填充了 Weak→strong 表格(仅空白;各行措辞未变)。未改动 settings 源,无需重新生成 schema。 变异探测每个探测都修改 SKILL.md、重跑聚焦套件、然后还原文件:
验证
未运行集成测试:本次改动是 skill 正文及其契约测试,聚焦 Vitest 套件已完全覆盖;不属于只能通过打包 CLI 或集成框架触达的行为。 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧵 Resolved all 7 selected review thread(s). · 已关闭全部选中的 7 条评审线程。 Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审;第 100 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
yiliang114
left a comment
There was a problem hiding this comment.
Approved at head 4a2c036. Contained change (+106/-16, 3 files): goal-draft guidance tightened toward concise, verifiable objectives, plus the last commit aligning the gate and budget default with the documented behavior. All checks green at head except review-pr, still running.
qwen-code-dev-bot
left a comment
There was a problem hiding this comment.
Approved at head 4a2c0363.
我上一张批准票挂在 a253680b 上,随着这次推送已被标记为过期,所以这是在新 head 上重投的一票。 增量只有一个提交、两个文件、+34/-14,内容是把 skill 的措辞对齐到它自己该遵守的契约,而不是加功能。
对齐方向我两边都读了,确实是对齐不是放宽。 SKILL.md:111 现在把"什么算 essential"列成三项——未知的成功判据、未验证的命令或输入路径、没定下来的 edit 还是 replace——这与本 PR 自己写进 docs/users/features/goals.md:70 的那句逐类对应;而旧写法是"只要还剩任何 <TODO: …> 或关键决定",跟同一份文件里的另一条规则(非必要默认值用 [ASSUMPTION] 标,<TODO> 只留给缺成功判据的情况)互相矛盾:按旧措辞,一个只带非必要 <TODO> 的草稿本该被拒绝交付,而按 marker 规则它又该正常交付。收窄成枚举是把这条自相矛盾消掉,方向是收紧口径而不是放宽闸门。
这条改动唯一可能开洞的地方,在同一个 hunk 里堵上了。 自检第 7 条从"Context 里的路径和命令都验证过或是 <TODO>"改成"未验证的路径或命令属于 essential,必须保持 <TODO: …>",所以不能再用"标个 [ASSUMPTION]"的方式把一个没验证过的命令带进可执行的交接里。Budget 那三处也自洽了:模板行、自检第 10 条、以及弱/强示例表——强例子现在带 Context: [ASSUMPTION] the 20-turn budget is the drafter's default, not the user's,不再出现"示例里写默认预算但不声明这是假设"这种自我打脸;同时正文没有把它说成运行时强制的轮数上限,和文档里"写进 objective 不会配置计时器或改动 token 预算"仍然一致。
守卫套件我重新逐条比过,不是只看它没红。 这个 head 上 SKILL.test.ts 里 58 个正向 toContain 字面量全部存在于 SKILL.md(重写后用脚本再核了一遍);顺序 pin 是升级不是削弱——gate 锚点跟着新句子搬了位置,另外补了两条切片断言:交付句只能在 ## Step 0 之前匹配到(因为 "Needs clarification" 现在同时出现在 gate 句里,全局 toContain 会串味),以及 gate 到交接之间那段必须含 success criterion。这两处都是我会在自己 review 里要求的形状。
之前那轮已核实的前提没有被这次推送改动:propose_goal 只在 resolveInteractionMode === 'interactive' 时注册(packages/core/src/config/config.ts:9442),所以 Web Shell/ACP 拿到可复制命令是事实而不是取舍;tokensUsed/tokenBudget 在 web-shell mapper 里走 getNumber + 可选展开、不进必填校验,这一点在 #11254 已核过,本 PR 不碰。
不阻塞、留个话头:同一页 goals.md:19 那句"footer pill 和每一张状态卡都显示花费"以及"Once a Goal has billed a turn"仍然不准确(legacy 状态卡不显示数字,实现判的是已记录花费),那是 #11248 带来的、本 PR 没动,值得顺手一个小 follow-up。
CI 事实。 这个 head 上 Test (ubuntu-latest) 24m15s、Lint & Static 13m55s、Integration Tests (no-AK) 10m13s 都绿,只有 review-pr 还在跑,没有任何红。页面仍是 REVIEW_REQUIRED/BLOCKED:还有 4 位被点名的真人 reviewer 没交票,机器人批准不满足这条要求——yiliang114 已经在这个 head 上 Approve 了,所以真正缺的是那四位里再补一张。
|
Post-merge review of head Verified
Executed: Cross-check: the earlier LGTM predates the alignment commit; nothing else open. Prompt-contract tests pin the text, not model compliance — as the PR itself discloses, including the E2E gap on the audit-quota behavior. No blockers found in what merged. Reviewed with AI assistance. |
What this PR does
Keeps
/goal-draftfocused on the reads needed to establish scope and verification, asks for compact completion checks, and explicitly allows an audit to find no defects. It preserves user requirements without adding arbitrary coverage or evidence quotas. Unresolved essential inputs produce a draft marked "Needs clarification", with no approval proposal or runnable Goal command.Default stopping assumptions are identified as model instructions rather than runtime-enforced limits. Ready drafts retain the existing terminal approval or manual-command handoff, with clear wording that the draft has not been applied.
Why it's needed
Drafting can expand into excessive investigation and add unsupported completion requirements, such as requiring an audit to find a defect. The previous fallback also allowed missing essential information to become assumptions in an actionable command. Users need a small, grounded objective and an accurate explanation of what remains unresolved before starting autonomous work.
Reviewer Test Plan
How to verify
/goal-draftto address a documented failing test. Expect a concise objective using the real verification command, with no test execution or implementation during drafting./goal set,/goal edit, or approval proposal.Evidence (Before & After)
Global CLI baseline samples added an unrequested "20 turns or 10 minutes" budget without identifying it as an execution convention. Three changed-bundle samples reached final output: the completed ready drafts marked the default budget as an assumption and said the draft was not applied; the missing-criterion sample returned TODOs without an actionable command. The audit sample timed out at both 120s and 240s, so zero-defect/coverage-quota behavior is not yet verified with real model output. Detailed results are in the separate E2E report comment.
Local validation passed: clean
npm ci, full build and bundle, full typecheck, 69 tests across the bundled skill and Goal tools, changed-test ESLint, changed-file Prettier, andgit diff --check. The bundled skill matches the edited source. Prompt-contract tests do not prove model compliance.Tested on
Environment (optional)
Node.js 24.14.1; global CLI baseline and local
node dist/cli.jsin isolated fixture/runtime directories. Model: qwen3.8-max.Risk & Scope
Linked Issues
None.
中文说明
此 PR 做了什么
让
/goal-draft只读取确定范围和验证方式所需的信息,生成简洁的完成条件,并明确允许审计未发现缺陷。保留用户要求,不额外添加覆盖量或证据数量指标。必要信息未解决时,只输出“待澄清”草稿,不发起目标批准,也不附可执行的 Goal 命令。默认停止条件会标记为假设,并说明它属于模型执行约定,不是运行时强制限制。完整草稿继续使用现有终端批准或手动命令交接,并明确草稿尚未应用。
为什么需要
草拟可能演变为过度调查,并附加缺乏依据的完成条件,例如要求审计必须发现缺陷。原有 fallback 还可能把缺失的必要信息当作假设写进可执行命令。用户需要在启动自主执行前,得到简洁、有依据的目标,并知道哪些信息尚未确定。
Reviewer 验证计划
如何验证
/goal-draft草拟修复已有文档说明的失败测试。应使用真实验证命令,目标简洁,草拟期间不运行测试或实现修复。/goal set、/goal edit,也不发起批准。修改前后证据
全局 CLI 基线样本增加了用户未要求的“20 回合或 10 分钟”预算,没有说明它只是执行约定。修改后 bundle 有三个样本得到最终输出:完整草稿标注了默认预算假设并说明尚未应用;缺失判据的样本只给 TODO,没有可执行命令。审计样本在 120 秒和 240 秒两次超时,因此零缺陷与覆盖配额的实际模型输出尚未验证。详细结果记录在单独的 E2E 报告评论中。
本地已通过:干净安装
npm ci、完整 build 和 bundle、完整 typecheck、bundled skill 与 Goal tools 共 69 个测试、修改测试文件的 ESLint、修改文件的 Prettier,以及git diff --check。打包 skill 与修改后的源码一致。提示词契约测试不能证明模型一定遵循指令。测试平台
环境
Node.js 24.14.1;全局 CLI 基线和本地
node dist/cli.js均在隔离的 fixture/runtime 目录运行。模型:qwen3.8-max。风险与范围
关联 Issue
无。