Skip to content

Outro names a report file the agent was told not to write #1332

Open

Description

@gewenyu99

Problem

Main now has part of the "report goes through publish_handoff" change, and the outros point at files that do not exist.

#1141 (merged 2026-08-27) added this clause to the publish_handoff description: "do not write the report to a file yourself". It is at src/agent/tools/handoff.ts:30-33 on main (d8486dc), and the pi tool uses the same text. The rest of the code still asks for the file or says it exists:

  • skillPrompt still says "write a brief markdown report to ./${reportFile}" (src/agent/agent-prompt.ts:54). Every program with a skillId gets this line.
  • The context-mill skills still say to write it. context/skills/data-warehouse-source/description.md:99 says "write the report file requested by the wizard". context/skills/audit/references/6-report.md:7 and :52 say to Write posthog-audit-report.md to disk. context-mill#306 moves these skills to publish_handoff. It is still a draft.
  • The outros show the file name and never check the file exists. src/agent/runner/sequence/linear.ts:294-300 passes run.reportFile through. src/ui/tui/screens/OutroScreen.tsx:87-93 prints "Check ./… for details", and src/ui/tui/exit-line.ts:91-92 adds the same line on exit. src/ui/tui/screens/audit/AuditOutroScreen.tsx:64-76 prints "Report saved to:". The audit success message at src/lib/programs/audit/index.ts:48, the ledger row label at src/lib/programs/audit/seed.ts:99-103 and the "Write report" slide at src/ui/tui/screens/audit/slides/writeReport.tsx:25 also name the file.
  • On main, publish_handoff writes a file only when the host sets POSTHOG_HANDOFF_OUTPUT_PATH. The TUI does not set it, so nothing is written to the project.
  • The wizard-workbench warehouse grader also expects the file. Its "report written" check passes only when result.reportFile.exists is true (PostHog/wizard-workbench services/wizard-ci/warehouse-checks.ts:813-820), so a run that follows the tool description fails it.

Evidence

Each run follows one instruction or the other. We saw this in a real-TUI sweep of all nine programs on the refactor stack head d938a43 (#1307), which has the same code for this path as main:

  • warehouse (revenue/stripe/stripe-saas-demo): the agent called only publish_handoff and marked "Write setup report" completed. The outro says "Check ./posthog-warehouse-report.md for details". No file was written.
  • audit (audit/posthog-demo-3000): the agent resolved write-report as pass: "Final markdown report delivered through the wizard handoff." The outro says "Report saved to: /posthog-audit-report.md". No file was written. The Release A top (200961f) did the same.
  • metrics (express-todo, orchestrator flow): the report went only to a notebook. The orchestrator outro does not name a file, so it shows nothing false.
  • ai-observability wrote its file and also called publish_handoff. On 200961f, warehouse wrote its file too. The result depends on which instruction the model follows.

The stack carries the same lines at other positions (src/agent/agent-prompt.ts:54, src/agent/tools/handoff.ts:33, src/agent/runner/sequence/linear.ts:314, src/ui/tui/screens/OutroScreen.tsx:107, src/ui/tui/screens/audit/AuditOutroScreen.tsx:67, src/programs/resolve-run-definition.ts:56), so this is not a stack regression.

Prior work

#1047 (open draft, last updated 2026-07-31, conflicting) makes publish_handoff the only report channel. It changes skillPrompt, stops programs naming a .md file, and shows the file in the outro only when one was written. context-mill#306 is its skill-side pair. Both are stale, and #1047 predates #1141, so it does not describe the contradiction main has now. Its base is ce8b70d, and the files have moved since: src/lib/agent is now src/agent, and src/lib/wizard-tools is now src/agent/tools.

#811 (open draft, stale since 2026-07-06) gates the linear-runner outro on the file existing. It is on old paths and does not cover the audit success message or screen.

Suggested fix

  1. Now: pick one instruction. Either drop "do not write the report to a file yourself" from PUBLISH_HANDOFF_DESCRIPTION until the rest moves, or change skillPrompt to order a publish_handoff call and stop naming the file.
  2. Until one channel remains, show a report path only when that file exists in installDir. fix(outro): only promise the setup report when it exists on disk #811 did this for the linear runner. The same check needs to cover the audit outro screen, the audit success message and the exit line.
  3. Longer term: land feat(wizard-tools): publish_handoff creates the notebook, outro opens it #1047 and context-mill#306 together, or close them and do the same change on the current paths. Update the workbench "report written" check to accept the published handoff when the program no longer names a file.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions