Repository navigation
feat(orchestrator): read the repackaged step skills behind wizard-agent-dag-skills - #1559
Open
edwinyjlim wants to merge 9 commits into
Open
edwinyjlim wants to merge 9 commits into
edwinyjlim wants to merge 9 commits into
Conversation
…nt-dag-skills With `wizard-agent-dag-skills` on, a task reads the skill that stands in for the one its prompt names: the menu entry whose `replaces` is that name. An entry with `frameworkReferences` is one skill written for every framework, and the runner completes it from the framework's reference skill: the product's pages and the rules file are copied into the skill, and SKILL.md gets the framework's name and the lines that list them. When a replacement fails to install, or the reference skill cannot complete it, the task reads the skill its prompt names, and the run sends `orchestrator repackaged skill unavailable` with the reason. The `orchestrator_skills` tag, set when the menu loads, says which a run read: `repackaged` when the flag is on and the menu has a replacement, `bundles` otherwise. A run pinned to a context-mill release reads no repackaged skill. `pnpm verify:repackaged-skills` checks a context-mill build or release: for every framework and every replaced skill, the files a task reads are the same both ways. Co-Authored-By: Claude <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
This was referenced Nov 13, 2026
Closed
The trait that holds a framework's variant is `frameworkVariant`. It reads the same value: the stable id a framework config returns from `getVariant`, such as `pages-router` or `expo`. Nothing reads the trait yet and it is not a match key, so no run changes. Co-Authored-By: Claude <noreply@anthropic.com>
posthog-security-review-bot
Bot
added
the
security-review
|
✅ Security review complete — 3 findings posted as a review, in 14 min. Add the |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| throw new Error(`a composed file escapes the skill: ${file}`); | ||
| } | ||
| fs.mkdirSync(path.dirname(target), { recursive: true }); | ||
| fs.writeFileSync(target, contents); |
Collaborator
There was a problem hiding this comment.
Can we guard against existing symlinks here? I reproduced this locally with a reference file symlinked to a file outside the skill folder, and composition overwrote that outside file with the pack content. readFiles skips the symlink, but writeFileSync follows it, so the path prefix check doesn't catch it. Could we reject symlinks in the output path or compose in a fresh private directory first?
| path.join(input.installDir, result.path), | ||
| path.join(input.installDir, referenceInstallPath), | ||
| product, | ||
| ); |
Collaborator
There was a problem hiding this comment.
Can we scan the completed skill before using it? The install scan happens before composition, but parsing variant.json can turn Unicode-escaped metadata into plaintext instructions in SKILL.md. I checked with the real Warlock scanner locally: the escaped JSON passed, while the decoded text triggered a critical prompt-injection match. We should scan after composition and discard the replacement / fall back if the scan rejects it or errors.
![posthog-security-review-bot[bot]](https://avatars.githubusercontent.com/in/4551195?s=60&v=4){kind=small}
posthog-security-review-bot
Bot
removed
the
security-review
sarahxsanders
approved these changes
Nov 13, 2026
johncwaters
reviewed
Nov 13, 2026
Contributor
There was a problem hiding this comment.
Verdict: APPROVE WITH NITS
See inline comments.
Automated review by Glimmervoid. Not written by a human
| packDir: string, | ||
| product: string, | ||
| ): void { | ||
| const stepFiles = readFiles(skillDir); |
Contributor
There was a problem hiding this comment.
[boundary] MEDIUM
A step-skill folder left over from an earlier run can beat the fresh pack pages. Only cleanupQueue removes <installDir>/.posthog-wizard-cache/skills/<replacement.id>, at the end of a run (orchestrator-runner.ts:702-707), and queue.ts:177-184 already allows for runs that end without it. downloadSkill unzips over the folder without clearing it (skill-install.ts:102), so this line reads the old composed references/<page>.md as one of the step skill's own files. composeStepSkill then merges stepFiles after the pack pages (line 102), the old copy wins, and line 148 skips the write because the contents match. The task ends up reading the previous release's or framework's page. The legacy path isn't affected.
Fix: rmSync the replacement folder in installReplacement before installSkillById, or compose from the archive's file list.
Automated review. Not written by a human
Completing a step skill from its pack checks every target before the first write. It refuses a file that would leave the install directory or pass through a link below it: a folder on the way that is a link, or a file that is one. A project can ship such links in the cache folder, and a write through one lands outside the skill. The orchestrator then scans the completed skill, since completing it writes text the install scan never read. A skill the scan rejects, or whose scan fails, is deleted, and the task reads the skill its prompt names. Co-Authored-By: Claude <noreply@anthropic.com>
The check compares a SKILL.md whole but for the `name` in its frontmatter, the one field a skill and its replacement must differ in. A skill family is compared once per framework even when no pack completes its replacement, and a replaced name with no skill to compare fails the check. Co-Authored-By: Claude <noreply@anthropic.com>
The cache folder is removed when a run ends, so a run that ends another way leaves its skills behind, and the installer unpacks over what is there. Completing a step skill takes every file in its folder as the skill's own, so a page the earlier run composed would replace the pack's page of the same name. The orchestrator now removes a replacement's folder before it installs the replacement. The removal reaches the folder through no link below the install directory, and a refusal sends the task to the skill its prompt names. Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
Workbench eval of
|
| App | Verdict | Tasks done, off / on | Difference between arms |
|---|---|---|---|
next-js/15-app-router-saas |
Pass | 9 / 9 | Off wires 13 server events, on wires 10. Both have 0 tsc errors |
rails/fizzy |
Pass | 9 / 9 | None beyond optional logs-other reads |
swift/hackers-ios |
Pass | 8 / 7 | Error tracking is "done" off and "not needed" on. Both set errorTrackingConfig.autoCapture = true in init |
android/Jetchat |
Pass | 9 / 8 | Identify is "done" off and "not needed" on, which looks like agent variance |
laravel/laravel12-saas |
Pass | 8 / 8 | None |
flutter/localsend |
No signal | 0 / 0 | Both arms abort with Could not detect a framework. The Flutter app sits in a subfolder |
- Task order. No task started before a task in its
dependsOnfinished, in either arm. Start order only differs where two tasks started in the same second, like install and init. - Docs read. Each step skill read the same reference pages in both arms. Every extra read was an optional skill the agent picked, like
posthog-best-practicesorlogs-*. - Fallbacks. No run logged
repackaged skill unavailable,reference unavailableorno integration skill. - Harness. The workbench
keep-skillsassertion fails on both arms, since the orchestrator flow goesoutro → mcp → exit. The skill cache is gone before the harness can hash it, so the staticverifyrun covers the skill trees.
Still running: Django, React Router SaaS template, SvelteKit CMSaasStarter, Nuxt 4, triplex and create-t3-turbo through posthog-integration, and error tracking on React Router and Nuxt 4. Their results go in a second comment.
Snapshots
Each app has the last run frame, which shows the final task list, and the outro for both arms.
pi__next-15-app-router-saas
Flag off
28-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Wire user identification
customer's situation. ◼ Add error tracking
◼ Capture events
◆ Get way more detail using properties ◼ Create a starter dashboard
Events and person records can have any properties you ◼ Add log capture
want. Track things like how they found your website, ◼ Verify and review the integration
what subscription tier they choose, and much more. ▶ Write the setup report
◆ Use PostHog in Slack ⠸ Progress: 8/9 completed
Connect the PostHog Slack app to analyze data and ship (1 skipped as not required)
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
◆ You can track Stripe revenue with PostHog
Add Stripe as a data source while you wait: https://app.
posthog.com/data-warehouse/new-source?kind=Stripe
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ The integration compiled and passed Next.js type validation; the build then stopped during page-data collection
because the unrelated `POSTGRES_URL` is absent. I’m checking whether the project has any configured scoped linter or
formatter before closing verification.
◆ Verification is complete: dependency installation and frozen reinstall passed; compilation and type validation
passed after the OpenTelemetry API fix; full page-data collection is blocked only by the project’s missing
`POSTGRES_URL`. No project linter or formatter is configured, so there were no scoped lint/format commands to run.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
29-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/63d39bf30830?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351592?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: `pnpm build` could not complete page-data collection because the pre-existing required
`POSTGRES_URL` environment variable is missing; integration compilation and type validation passed.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Query your other data in PostHog:
• Connect Postgres:
https://us.posthog.com/project/228144/data-warehouse/ne…
• Connect Stripe:
https://us.posthog.com/project/228144/data-warehouse/ne…
• Connect them all at once with: npx @posthog/wizard warehouse
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
Flag on
28-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Wire user identification
customer's situation. ◼ Add error tracking
◼ Capture events
◆ Get way more detail using properties ◼ Create a starter dashboard
Events and person records can have any properties you ◼ Add log capture
want. Track things like how they found your website, ◼ Verify and review the integration
what subscription tier they choose, and much more. ▶ Write the setup report
◆ Use PostHog in Slack ⠸ Progress: 8/9 completed
Connect the PostHog Slack app to analyze data and ship (1 skipped as not required)
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
◆ You can track Stripe revenue with PostHog
Add Stripe as a data source while you wait: https://app.
posthog.com/data-warehouse/new-source?kind=Stripe
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ Checking whether the project defines scoped lint or format tooling for the changed files.
◆ Finalizing the verification handoff with the completed review, install evidence, and the remaining post-fix
build-verification limitation.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
29-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/8ae8337c0db8?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351638?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Verification incomplete: the final allowed `pnpm build` attempt failed in `instrumentation.ts`
before the exact reported constructor-shape fix was applied, and no post-fix build rerun was permitted.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Query your other data in PostHog:
• Connect Postgres:
https://us.posthog.com/project/228144/data-warehouse/ne…
• Connect Stripe:
https://us.posthog.com/project/228144/data-warehouse/ne…
• Connect them all at once with: npx @posthog/wizard warehouse
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
pi__rails-fizzy
Flag off
29-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Wire user identification
customer's situation. ◼ Add error tracking
◼ Capture events
◆ Get way more detail using properties ◼ Create a starter dashboard
Events and person records can have any properties you ◼ Add log capture
want. Track things like how they found your website, ◼ Verify and review the integration
what subscription tier they choose, and much more. ▶ Write the setup report
◆ Use PostHog in Slack ⠙ Progress: 8/9 completed
Connect the PostHog Slack app to analyze data and ship (1 skipped as not required)
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ Lint also cannot start because Bundler 2.7.2 is unavailable. I’m checking lockfile package entries without reading
the full lockfile, then I’ll finalize with exact install/build/lint limitations and review results.
◆ Review is complete: all four dimensions pass after three targeted fixes, with event names, properties, and firing
locations unchanged. Installation and automated checks remain blocked by the project-pinned Bundler version and the
runtime’s Rails-command restriction.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
30-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/b618bf5aa41c?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351686?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Verification incomplete: Bundler 2.7.2 is unavailable, `bundle add` and scoped RuboCop could not
start, and the runtime blocked `bin/rails zeitwerk:check`; Gemfile.lock was not updated.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Query your other data in PostHog:
• Connect MySQL:
https://us.posthog.com/project/228144/data-warehouse/ne…
• Connect them all at once with: npx @posthog/wizard warehouse
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
Flag on
29-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Wire user identification
customer's situation. ◼ Add error tracking
◼ Capture events
◆ Get way more detail using properties ◼ Create a starter dashboard
Events and person records can have any properties you ◼ Add log capture
want. Track things like how they found your website, ◼ Verify and review the integration
what subscription tier they choose, and much more. ▶ Write the setup report
◆ Use PostHog in Slack ⠸ Progress: 8/9 completed
Connect the PostHog Slack app to analyze data and ship (1 skipped as not required)
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ CI confirms RuboCop is the project lint check and defines no separate formatter. I’m inspecting the CI task list
only to confirm whether it includes a non-test build/typecheck stage.
◆ Verification is complete to the extent the environment allows: install and scoped lint were blocked by missing
Bundler 2.7.2, the Rails compile command was blocked before execution, and the project defines neither a separate
typecheck nor format check. I’m handing off the reviewed integration with the exact unresolved verification
conflict.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
30-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/ce69e5eee798?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351712?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Verification incomplete: Bundler 2.7.2 is missing (blocking install and lint), and the runtime
blocked `bin/rails zeitwerk:check` before execution; Gemfile.lock remains unresolved.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Query your other data in PostHog:
• Connect MySQL:
https://us.posthog.com/project/228144/data-warehouse/ne…
• Connect them all at once with: npx @posthog/wizard warehouse
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
pi__swift-hackers-ios
Flag off
27-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Add error tracking
customer's situation. ◼ Capture events
◼ Create a starter dashboard
◆ Get way more detail using properties ◼ Add log capture
Events and person records can have any properties you ◼ Verify and review the integration
want. Track things like how they found your website, ▶ Write the setup report
what subscription tier they choose, and much more.
⠋ Progress: 7/8 completed
◆ Use PostHog in Slack (2 skipped as not required)
Connect the PostHog Slack app to analyze data and ship
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ The simulator build and integrated typecheck passed. I’m running SwiftLint and SwiftFormat checks only on the two
changed Swift files.
◆ Build verification is complete. The repository’s lint and format configurations exist, but the runtime blocked
both scoped tool invocations before execution; I’m recording that verification limitation precisely in the handoff.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
28-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/5807d06c6605?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351751?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Scoped lint and format verification are incomplete: the runtime command allowlist blocked both
`swiftlint` and `swiftformat` before execution.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
Flag on
27-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Capture events
customer's situation. ◼ Create a starter dashboard
◼ Add log capture
◆ Get way more detail using properties ◼ Verify and review the integration
Events and person records can have any properties you ▶ Write the setup report
want. Track things like how they found your website,
what subscription tier they choose, and much more. ⠴ Progress: 6/7 completed
(3 skipped as not required)
◆ Use PostHog in Slack
Connect the PostHog Slack app to analyze data and ship
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ The full app build and Swift typecheck passed. I’m running SwiftLint only on the seven changed Swift
source/manifest files.
◆ Build verification passed. The scoped SwiftLint invocation was blocked by the runtime before execution, so lint is
unverified; the project has no runnable format-check script, and I’m finalizing with that limitation recorded.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
28-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/c0868d3c225a?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351783?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Scoped SwiftLint verification is incomplete because the runtime blocked the configured `swiftlint`
executable before execution.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
pi__android-jetchat
Flag off
27-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Wire user identification
customer's situation. ◼ Add error tracking
◼ Capture events
◆ Get way more detail using properties ◼ Create a starter dashboard
Events and person records can have any properties you ◼ Add log capture
want. Track things like how they found your website, ◼ Verify and review the integration
what subscription tier they choose, and much more. ▶ Write the setup report
◆ Use PostHog in Slack ⠋ Progress: 8/9 completed
Connect the PostHog Slack app to analyze data and ship (1 skipped as not required)
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ The harness does not permit the project’s Spotless check task, so formatting could not be executed. I’m doing a
final static contract check for singleton setup, guarded SDK access, event names, and non-PII properties before
handing off.
◆ Static review confirms one Application-level setup, all SDK singleton access guarded after successful setup,
unchanged event contracts, fixed bounded properties, and the documented Logs/error configuration. I’m completing
with the Java/tooling verification limitations called out explicitly.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
28-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/a79f6351b7ea?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351824?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Verification incomplete: dependency resolution, build, and lint could not start because no Java
runtime is installed; the Spotless format-check task was blocked by the command allowlist.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
Flag on
27-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Add error tracking
customer's situation. ◼ Capture events
◼ Create a starter dashboard
◆ Get way more detail using properties ◼ Add log capture
Events and person records can have any properties you ◼ Verify and review the integration
want. Track things like how they found your website, ▶ Write the setup report
what subscription tier they choose, and much more.
⠹ Progress: 7/8 completed
◆ Use PostHog in Slack (2 skipped as not required)
Connect the PostHog Slack app to analyze data and ship
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ Android lint was likewise blocked before execution by the missing Java runtime; I’m making the final Spotless
format-check attempt for the app module.
◆ Verification is complete to the environment’s limit: review passed with one minimality fix, while dependency
resolution, build/typecheck, and lint were blocked by the absent Java runtime, and the harness disallowed the
project’s Spotless task.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
28-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/45c4a156b68d?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351850?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Verification incomplete: Java is unavailable for dependency resolution/build/typecheck/lint, and the
runtime allowlist blocked `./gradlew :app:spotlessCheck`.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
pi__laravel12-saas
Flag off
31-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Wire user identification
customer's situation. ◼ Add error tracking
◼ Capture events
◆ Get way more detail using properties ◼ Create a starter dashboard
Events and person records can have any properties you ◼ Verify and review the integration
want. Track things like how they found your website, ▶ Write the setup report
what subscription tier they choose, and much more.
⠋ Progress: 7/8 completed
◆ Use PostHog in Slack (2 skipped as not required)
Connect the PostHog Slack app to analyze data and ship
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ Linting each changed PHP and Blade file with PHP’s syntax analyzer; no project typecheck, lint script, or format
script is defined.
◆ Finalizing the verification handoff with the successful build, the single review fix, and the Composer/PHP checks
that this environment could not run.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
32-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/978ed4b9a1b6?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351874?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Composer is unavailable, so the PHP dependency, Artisan package installation/discovery, and PHP
lint/format checks remain unverified; `npm run build` passed.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Query your other data in PostHog:
• Connect Stripe:
https://us.posthog.com/project/228144/data-warehouse/ne…
• Connect them all at once with: npx @posthog/wizard warehouse
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
Flag on
31-run.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
Tips Tasks
◆ You can also track people and groups with PostHog ◼ Add the PostHog SDK to the manifest
Events can be associated with the humans who generate ◼ Set up PostHog initialization
them, letting you understand a specific user or ◼ Wire user identification
customer's situation. ◼ Add error tracking
◼ Capture events
◆ Get way more detail using properties ◼ Create a starter dashboard
Events and person records can have any properties you ◼ Verify and review the integration
want. Track things like how they found your website, ▶ Write the setup report
what subscription tier they choose, and much more.
⠼ Progress: 7/8 completed
◆ Use PostHog in Slack (2 skipped as not required)
Connect the PostHog Slack app to analyze data and ship
product changes — deploy flags, open PRs, run queries —
just by tagging @PostHog: https://posthog.com/slack
◆ Chat with other builders
Introduce yourself on the forum at
https://posthog.com/forum, or see what other builders
are working on in Discord while you wait:
https://discord.gg/xBXBD8edK3
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
┊ The Vite production build passed. With no Composer/vendor tooling available and no separate typecheck, lint, or
format scripts defined, I’m using PHP’s native syntax linter on every changed PHP/Blade file.
◆ Verification is complete: the frontend build passed, while PHP dependency installation and PHP lint/format checks
could not run because Composer is unavailable and the runtime blocks direct `php -l`; no retry is appropriate.
Status Tail logs HN Visualizer
←→ switch tab s toggle status
32-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✔ PostHog set up, with one conflict to review.
The wizard has set up PostHog in your codebase and written a full report of what it instrumented. Read the notebook
for more details!
Notebook: https://us.posthog.com/project/228144/notebooks/94073858227a?utm_source=wizard&utm_medium=cli&utm_content=ou
tro-notebook
Dashboard:
https://us.posthog.com/project/228144/dashboard/2351887?utm_source=wizard&utm_medium=cli&utm_content=outro-dashboard
⚠ Build conflict: Composer is unavailable, so the Laravel SDK is not installed or locked and PHP package-discovery,
lint, and format verification remain incomplete; `npm run build` passed.
Full details are in the setup report.
Learn more: https://posthog.com/docs/ai-engineering/ai-wizard?utm_source=wizard&utm_medium=cli&utm_content=outro-docs
Query your other data in PostHog:
• Connect Stripe:
https://us.posthog.com/project/228144/data-warehouse/ne…
• Connect them all at once with: npx @posthog/wizard warehouse
Note: This wizard uses an LLM agent to analyze and modify your project. Please review the changes made.
How did this work for you? Drop us a line: wizard@posthog.com
Press any key to continue
pi__flutter-localsend
Flag off
02-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✘ Could not detect a framework
Run the wizard from an app root directory.
That's the folder containing package.json or an equivalent project file.
Supported frameworks and manual setup: https://posthog.com/docs/libraries
Press any key to exit
Flag on
02-outro.ans
PostHog Wizard v2.94.0 Feedback: wizard@posthog.com
✘ Could not detect a framework
Run the wizard from an app root directory.
That's the folder containing package.json or an equivalent project file.
Supported frameworks and manual setup: https://posthog.com/docs/libraries
Press any key to exit
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
Collaborator
Full snapshots:
|
gewenyu99
reviewed
Nov 13, 2026
gewenyu99
left a comment
gewenyu99
left a comment
Collaborator
There was a problem hiding this comment.
🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.
… a repackaged skill The install of a repackaged skill moves out of the orchestrator into repackaged-skills.ts, with a test for each way it ends. - A replacement's id comes from the menu and names the folder that is emptied before the install. An id that is not one plain name, such as `../../src`, is refused before anything is removed or installed, and only a folder directly under the skills cache is ever removed. - Any failure of the install, a thrown error included, sends every task that waits on it to the skill its prompt names. A delete that fails no longer fails those tasks. - A pack page is copied only when every part of its path is a plain name and it ends in `.md`. The install scan reads lowercase extensions and skips dotfiles, so any other page would reach a task unscanned. - Completing a skill rewrites SKILL.md and copies pages the pack's install already scanned, so only SKILL.md is scanned again. An unreadable file fails that scan. - A `variant.json` with no `rules` reads as one with none, as context-mill reads it. Co-Authored-By: Claude <noreply@anthropic.com>
`pnpm verify:repackaged-skills <source> --record` writes one matching pair into fixtures/context-mill/repackaged: the step skill as published, the files of the pack that completing it reads, and the skill it replaces. A unit test completes that step from that pack and expects that skill, so the wizard's fill stays in step with context-mill's in CI. The pair is `integration-v2-init` on Flask, recorded from a build of PostHog/context-mill#529. The menu recorder now replaces only what it records, so it leaves the pair alone. Co-Authored-By: Claude <noreply@anthropic.com>
`--record` checks every destination against its folder of the fixture before it replaces the fixture or writes a file. A published file name such as `../../../../package.json` is refused, and nothing is written. Co-Authored-By: Claude <noreply@anthropic.com>
The check's comparison moves into `@shared/skill-composition` as `compareSkillFiles`, with tests on pairs that differ. The script and the recorded pair's test both call it, so what "reads the same" means is tested and written once. The script reports what it reported before. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With
wizard-agent-dag-skillson, a task reads the repackaged step skill that stands in for the one its prompt names, completed from the framework's pack. With the flag off or unset, a run reads the skills it reads today.The flag is independent of
wizard-agent-dag, and it does nothing until a context-mill release publishes the repackaged skills.Stack
The second half of the migration, behind
wizard-agent-dag-skills. The first half, behindwizard-agent-dag, is in context-mill 1.65.0 (PostHog/context-mill#447) and wizard 2.94.0 (#1551 to #1554).variant.json.main. Adds files and tags, and writes both pages where one replaced another.integration-<step>.main, on its own. Documentation only. Merges whenwizard-agent-dagreaches 100%.wizard-agent-dag-skillson, a task reads the step skill from B, completed from the pack from A.main. Does nothing while the flag is off.A and B are one GitHub stack and merge together. Apart from the page fix in A, they only add files, skills and tags, and no run reads those until the flag is on.
Rollout
wizard-agent-dag-skillsis a percentage flag, independent ofwizard-agent-dag. Turning it off returns every run to the skills it reads today, and a run pinned to a context-mill release reads those throughout.check:bundle-diff --additiveandcheck:repackaged-skillson the release buildpnpm verify:repackaged-skillsagainst the release, and a workbench run with the flag on compared with one with it off.orchestrator_skillstag: run success, task failures, and anyorchestrator repackaged skill unavailableC merges on its own, when
wizard-agent-dagreaches 100%.How a task gets its skill
flowchart TD name["A prompt names a skill,<br/>such as integration-v2-init"] --> flag{"wizard-agent-dag-skills on?"} flag -- no --> today["Install the skill the prompt names"] flag -- yes --> entry{"A menu entry whose<br/>replaces is that name?"} entry -- no --> today entry -- yes --> empty["Check the entry's id,<br/>then empty its folder"] empty --> install["Install the entry"] install --> refs{"frameworkReferences set?"} refs -- no --> done["The task reads it"] refs -- yes --> complete["Complete it from the<br/>framework's reference skill"] complete --> scan["Scan the SKILL.md it rewrote"] scan --> done empty -- "refused" --> fallback["Send orchestrator repackaged skill unavailable"] install -- "fails" --> fallback complete -- "fails" --> fallback scan -- "rejects it, or fails" --> fallback fallback --> todayA replacement installs once a run, so no task reads one that another is still completing. The install lives in
orchestrator/repackaged-skills.ts, and any failure in it, a thrown error included, sends the task to the skill its prompt names.What the wizard reads from context-mill
replacesframeworkReferencesvariant.jsonSKILL.md{framework_name},{framework_docs},{framework_rules}@shared/skill-compositioncopies the product's pages and the rules file into the skill'sreferences/and fills the three placeholders. No skill or product name is in the wizard: the menu says what replaces what.What a menu, a pack and a project's files cannot do
A replacement's id comes from the menu, its pages come from a pack, and the skills live in the project's
.posthog-wizard-cache/, which a project can ship and an earlier run can leave behind.../../src, and the run empties the folder that id namespage.MDor a dotfile.mdvariant.jsonSKILL.mdit rewrote is scanned. A skill the scan rejects, or whose scan fails, is deletedEach refusal sends the task to the skill its prompt names.
When a replacement cannot be used
variant.jsonor lacks the productThe preflight still requires every skill a prompt names, so the fallback is always there.
Analytics
orchestrator_skillstag, on every event sent after the menu loads:repackagedwhen the flag is on and the menu has a replacement,bundlesotherwise. A run against a release without the repackaged skills reportsbundleswhatever the flag says.orchestrator repackaged skill unavailable: the replacement, the name it replaces, the reason and the framework.The trait's name
The trait that holds a framework's variant is
frameworkVariant, the key that PostHog/context-mill#528's rows and tags use: a pack written for one variant carries aframeworkVariant:<value>tag. It reads the same value as before, the id a framework config returns fromgetVariant. Nothing reads the trait yet and it is not a match key, so no run changes.Proof that a task reads the same documents
pnpm verify:repackaged-skills [dist-or-url]checks a context-mill build or release. For every framework and every replaced skill, it resolves both skills as a run does, completes the replacement from the framework's pack, and compares the files a task reads: the same paths with the same content hash, and the sameSKILL.mdbut for thenamein its frontmatter. A skill family is compared once per framework, and a replaced name with no skill to compare fails the check.Against a build of PostHog/context-mill#529: 176 skill and framework pairs across 28 frameworks, and all 176 read the same documents.
The comparison is
compareSkillFilesin@shared/skill-composition, with its own tests.One pair is recorded in
fixtures/context-mill/repackaged/, with--record: the step skill as published, the files of the pack that completing it reads, and the skill it replaces. A unit test completes that step from that pack and expects that skill, so CI catches the wizard's fill drifting from context-mill's.Nuxt
Nuxt is the one framework whose skill families disagree on a default: the packs default to
nuxt-3-6, and the error tracking step tonuxt-4. So a Nuxt run reads the 3.7 error tracking page today. PostHog/context-mill#528'snuxt-3-6row lists that page, and a Nuxt run reads the same page with the flag on.Test plan
pnpm typecheckpnpm lintpnpm exec vitest run: 292 files passpnpm buildtraits.test.ts: the trait reads the same values under its new nameskill-composition.test.ts: the completed files, the order in which same-named files replace each other, each way a pack can fail to complete a skill, page names the install scan would skip, ids that are not one plain name, and no write or removal through a linked file, a linked folder or a linked skill folderrepackaged-skill-installer.test.ts: each way an install ends, among them a menu id that names another folder, an install that throws, a missing reference skill, a linked skills folder, and a rejected or failed scanrepackaged-skills.test.ts: a full orchestrator run with the flag on, with it off, with a menu that has no replacement, with fetched prompts, with a page left by an earlier run, with no reference skill, with a linked skills folder, and with each other fallbackyara-hooks.test.ts: the scan of one file refuses a critical match, reads that file alone, and fails on a file it cannot readskill-composition-fixture.test.ts: the recorded step, completed from the recorded pack, is the recorded skill it replacesskill-composition.test.ts:compareSkillFiles, the comparison the check runs, on a replacement that differs in its body, its description, a page or its file listpnpm verify:repackaged-skillsagainst a build of feat(integration): publish each step skill once, completed from the framework's pack context-mill#529: all 176 pairs read the same documents🤖 Generated with Claude Code