Repository navigation
feat(integration): publish each orchestrator step as one skill - #537
Conversation
A step of the orchestrator flow is one skill, `integration-<step>`, with its template in its own folder under `context/skills/integration/`. The per-framework step skills under `integration-v2` and `replay-vision-enable-replay` are no longer built, and a menu entry carries no `replaces`. The Replay vision skill names the step skills it hands its agent to. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a wizard branch:
Add Results will be posted here when complete. |
johncwaters
left a comment
There was a problem hiding this comment.
Verdict: APPROVE WITH NITS
See inline comments.
Automated review by Glimmervoid. Not written by a human
| | skill id | owns | | ||
| | --- | --- | | ||
| | `replay-vision-enable-replay` | making session replay record (both halves) | | ||
| | `integration-session-replay` | making session replay record (both halves) | |
There was a problem hiding this comment.
[code] MEDIUM
More a question than a bug. This tells the agent to install_skill integration-session-replay, integration-install and integration-init, which now ship with {framework_name} and {framework_docs} placeholders. The wizard's install_skill completes them from the pack as of PostHog/wizard#1566, so wizard runs are fine. What does the MCP server's install path do with them? If it hands them over as-is, the agent gets a skill with no framework pages, where replay-vision-enable-replay used to carry the real ones.
Automated review. Not written by a human
There was a problem hiding this comment.
Answering my own question with a run. Every consumer except the wizard gets the form, blanks included: integration-session-replay.zip from this stack's build opens with # Make session replay record — {framework_name}, and its only reference is the generic installation.md. The replay-vision-enable-replay bundle it replaces shipped one variant per framework.
The fill exists twice: scripts/lib/skill-composition.js here, which only its test calls, and the TypeScript port in PostHog/wizard#1566, which is the one that runs. I ran both over every step skill and pack in the build: 6 steps, 40 packs, 240 pairs, 0 outputs that differ. So the two agree today, but no test compares them, and the wizard's fixture is recorded from its own port.
Since they agree, collapsing them costs nothing. Could context-mill publish the filler as a small package that the wizard and the MCP server both import, and delete the port? Then install_skill from MCP hands back a filled skill, and there's one place the format can change.
Repro
unzip -p dist/skills/integration-session-replay.zip SKILL.md | grep -n '{framework_'
# 3:description: Make session replay record for {framework_name}
# 9:# Make session replay record — {framework_name}probe2.ts, run with tsx in a wizard#1568 checkout: for each menu entry with frameworkReferences and each integration-* zip that has a variant.json listing that product, it calls the wizard's composeStepSkill(stepFiles, packFiles, product) and this repo's composeStepSkill({ stepFiles, packFiles, product }) and compares the outputs.
steps 6, packs 40, pairs compared 240, outputs differing 0
There was a problem hiding this comment.
its a good callout. we need to rethink how skill delivery to the MCP server and plugins work. most of these skills are only relevant to the wizard harness and its task structure too
gewenyu99
left a comment
There was a problem hiding this comment.
🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.
| shared_docs: | ||
| - https://posthog.com/docs/session-replay/installation.md | ||
| replaces: replay-vision-enable-replay | ||
| framework_references: integration |
There was a problem hiding this comment.
Hey, this template doesn't contain {references}, so I think the framework docs and rules get dropped silently. The other five step templates (capture, error-tracking, feature-flags, init, install) all have it, and the last line here still tells the agent to read "the framework docs below".
Adding ## Reference and {references} at the end of description.md should fix it.
| type: docs-only | ||
| template: ../../replay-vision-enable-replay/description.md | ||
| template: description.md | ||
| description: Make session replay record for {display_name} |
There was a problem hiding this comment.
I think this will show {framework_name} as-is in skill-menu.json, because:
framework_referencesis set, soexpandSkillGroupsreplaces{display_name}with{framework_name}(skill-generator.js:447){framework_name}is only filled in bycomposeStepSkillwhen a pack is installed- menu,
cliEntries, andmanifest.jsonentries never go throughcomposeStepSkill
| description: Make session replay record for {display_name} | |
| description: Make session replay record on server and client |
|
|
||
| - From the project state / MCP tools, note: is session replay recording on, does the project have recordings, and what scanners already exist (`vision-scanners-list` via the PostHog `exec` tool). | ||
| - From the repo: is PostHog integrated (`posthog-js` / a server SDK in the dependency manifests, a snippet, or events in the project state)? **If it is not, do not stop** - install and initialize the SDK first, following the `integration-v2-install` and `integration-v2-init` skills (install each with `install_skill`). The minimal SDK footprint they leave behind is enough for recordings to flow; skip identify, capture, and dashboards - this run sets up Replay vision, not the full integration. | ||
| - From the repo: is PostHog integrated (`posthog-js` / a server SDK in the dependency manifests, a snippet, or events in the project state)? **If it is not, do not stop** - install and initialize the SDK first, following the `integration-install` and `integration-init` skills (install each with `install_skill`). The minimal SDK footprint they leave behind is enough for recordings to flow; skip identify, capture, and dashboards - this run sets up Replay vision, not the full integration. |
There was a problem hiding this comment.
nit: I think it's possible for a caller without an integration pack (for example wizard skill replay-vision-setup or an external MCP client) to follow this and hit no framework pack, or see raw {framework_docs} placeholders. It may not happen today, but maybe guard against it, for example by telling non-orchestrator callers to load the integration family and install the matching framework pack first.
| expect(group.variants).toEqual(replaced.variants); | ||
| it.each(steps)('%s keeps its template in its own folder', (key, group) => { | ||
| expect(group.template).toBe('description.md'); | ||
| expect(fs.existsSync(path.join(REPO_CONFIG_DIR, 'skills', ...key.split('/'), group.template))).toBe(true); |
There was a problem hiding this comment.
nit: this suite now only checks that each template exists, so the missing {references} and leaked {framework_name} above pass CI. Maybe assert {references} in each framework_references template, forbid {display_name} in its description, and compose one real step against a pack.
…elete-old-step-skills
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…it a plain description
The template named framework docs below it and listed none, so a completed
skill carried the framework's pages without naming them. Its description
held {display_name}, which the menu showed as a placeholder.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

What this does
A step of the orchestrator flow is one skill, written once:
integration-<step>.After this change a release has:
integration-install,integration-initand the rest. 13 are unchanged.integration-session-replaylists its references under a## Referenceheading, like the other five steps completed from a pack, and its description names no frameworkintegration-<variant>, unchangedintegration-v2-*skill: 48 zips and 4 group JSON files are gonereplay-vision-enable-replay.jsonreplaceson a menu entryHow the sources are laid out:
context/skills/integration/<step>/description.md. The 14 templates moved there fromcontext/skills/integration-v2/<step>/andcontext/skills/replay-vision-enable-replay/, and each config'stemplateisdescription.md.context/skills/integration-v2/andcontext/skills/replay-vision-enable-replay/are gone.framework_referencesis validated on its own.scripts/check-repackaged-skills.js, its package script and its CI step are gone, withcompareSkills.composeStepSkillstays as the reference for completing a step skill from a pack.One skill's text changes
context/skills/replay-vision/description.md, published asreplay-vision-setup, tells its agent to install three step skills by name. It now names the ones that publish:replay-vision-enable-replayintegration-session-replayintegration-v2-installintegration-installintegration-v2-initintegration-initThose three are written once for every framework, with placeholders for the framework's name, pages and rules. A consumer that installs one outside the orchestrator has to complete it from the framework's pack the same way.
Verification
npm test: 391 tests pass.integration-v2-*.zip, 4integration-v2-*.jsonandreplay-vision-enable-replay.json.integration-<step>.ziphold the same files with the same bytes.integration-session-replay.zipdiffers inSKILL.mdalone: its description and the reference list at its end.skill-menu.jsongoes from 307 entries to 254. The 14 step entries are the same but forreplacesand the name and description ofintegration-session-replay, and no other entry differs.cliEntriesloses the 248internalentries of the removed skills.manifest.jsonandskills-mcp-resources.ziplose the 48 removed skills, and the archive loses the 5 group JSON files.replay-vision-setup.zipdiffers in the three lines of itsSKILL.mdabove.Rollout
Stacked on the prompt removal.
A wizard whose prompts name
integration-v2-*orreplay-vision-enable-replayfinds no skill under those names once this ships, and its run fails at preflight. That is every wizard release before the one whose prompts name theintegration-<step>skills. Merge it only when all of these hold:integration-<step>skillswizard-agent-dagandwizard-agent-dag-skillsare both still on🤖 Generated with Claude Code