Skip to content

feat(integration): publish each orchestrator step as one skill - #537

Merged
edwinyjlim merged 5 commits into
dag/m1-delete-promptsfrom
dag/m2-delete-old-step-skills
Nov 17, 2026
Merged

edwinyjlim merged 5 commits into
dag/m1-delete-promptsfrom
dag/m2-delete-old-step-skills

Conversation

@edwinyjlim

@edwinyjlim edwinyjlim commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

What this does

A step of the orchestrator flow is one skill, written once: integration-<step>.

After this change a release has:

  • the 14 step skills, integration-install, integration-init and the rest. 13 are unchanged. integration-session-replay lists its references under a ## Reference heading, like the other five steps completed from a pack, and its description names no framework
  • the 40 framework packs, integration-<variant>, unchanged
  • no integration-v2-* skill: 48 zips and 4 group JSON files are gone
  • no replay-vision-enable-replay.json
  • no replaces on a menu entry

How the sources are laid out:

  • Each step skill keeps its template in its own folder, context/skills/integration/<step>/description.md. The 14 templates moved there from context/skills/integration-v2/<step>/ and context/skills/replay-vision-enable-replay/, and each config's template is description.md.
  • context/skills/integration-v2/ and context/skills/replay-vision-enable-replay/ are gone.
  • framework_references is validated on its own. scripts/check-repackaged-skills.js, its package script and its CI step are gone, with compareSkills. composeStepSkill stays as the reference for completing a step skill from a pack.

One skill's text changes

context/skills/replay-vision/description.md, published as replay-vision-setup, tells its agent to install three step skills by name. It now names the ones that publish:

Was Is
replay-vision-enable-replay integration-session-replay
integration-v2-install integration-install
integration-v2-init integration-init

Those three are written once for every framework, with placeholders for the framework's name, pages and rules. A consumer that installs one outside the orchestrator has to complete it from the framework's pack the same way.

Verification

  • npm test: 391 tests pass.
  • The build passes.
  • A build of this branch was compared with a build of the branch below it, both from a clean checkout:
    • 53 files are gone: 48 integration-v2-*.zip, 4 integration-v2-*.json and replay-vision-enable-replay.json.
    • 13 of the 14 integration-<step>.zip hold the same files with the same bytes. integration-session-replay.zip differs in SKILL.md alone: its description and the reference list at its end.
    • skill-menu.json goes from 307 entries to 254. The 14 step entries are the same but for replaces and the name and description of integration-session-replay, and no other entry differs. cliEntries loses the 248 internal entries of the removed skills.
    • manifest.json and skills-mcp-resources.zip lose the 48 removed skills, and the archive loses the 5 group JSON files.
    • replay-vision-setup.zip differs in the three lines of its SKILL.md above.
    • The marketplace tree is unchanged.

Rollout

Stacked on the prompt removal.

A wizard whose prompts name integration-v2-* or replay-vision-enable-replay finds no skill under those names once this ships, and its run fails at preflight. That is every wizard release before the one whose prompts name the integration-<step> skills. Merge it only when all of these hold:

  • cloud runs a wizard release whose prompts name the integration-<step> skills
  • older wizard versions have drained from the run data
  • wizard-agent-dag and wizard-agent-dag-skills are both still on
  • the marketplace and MCP server owners have agreed. The MCP server's archive carries the 48 removed skills today

🤖 Generated with Claude Code

A step of the orchestrator flow is one skill, `integration-<step>`, with
its template in its own folder under `context/skills/integration/`. The
per-framework step skills under `integration-v2` and
`replay-vision-enable-replay` are no longer built, and a menu entry carries
no `replaces`.

The Replay vision skill names the step skills it hands its agent to.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@edwinyjlim
edwinyjlim added this pull request to stack #539 November 17, 2026 01:10
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci feature-flags
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci feature-flags/django
  • /wizard-ci feature-flags/next-js
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/go-sdk
  • /wizard-ci mcp-analytics/python-sdk
  • /wizard-ci mcp-analytics/ruby-sdk
  • /wizard-ci mcp-analytics/rust-sdk
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a wizard branch:

  • /wizard-ci all wizard:my-branch

Add wizard:<branch> to any command above to pin the wizard branch. It defaults to main.

Results will be posted here when complete.

@johncwaters johncwaters left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE WITH NITS

See inline comments.

Automated review by Glimmervoid. Not written by a human

| skill id | owns |
| --- | --- |
| `replay-vision-enable-replay` | making session replay record (both halves) |
| `integration-session-replay` | making session replay record (both halves) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[code] MEDIUM

More a question than a bug. This tells the agent to install_skill integration-session-replay, integration-install and integration-init, which now ship with {framework_name} and {framework_docs} placeholders. The wizard's install_skill completes them from the pack as of PostHog/wizard#1566, so wizard runs are fine. What does the MCP server's install path do with them? If it hands them over as-is, the agent gets a skill with no framework pages, where replay-vision-enable-replay used to carry the real ones.

Automated review. Not written by a human

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answering my own question with a run. Every consumer except the wizard gets the form, blanks included: integration-session-replay.zip from this stack's build opens with # Make session replay record — {framework_name}, and its only reference is the generic installation.md. The replay-vision-enable-replay bundle it replaces shipped one variant per framework.

The fill exists twice: scripts/lib/skill-composition.js here, which only its test calls, and the TypeScript port in PostHog/wizard#1566, which is the one that runs. I ran both over every step skill and pack in the build: 6 steps, 40 packs, 240 pairs, 0 outputs that differ. So the two agree today, but no test compares them, and the wizard's fixture is recorded from its own port.

Since they agree, collapsing them costs nothing. Could context-mill publish the filler as a small package that the wizard and the MCP server both import, and delete the port? Then install_skill from MCP hands back a filled skill, and there's one place the format can change.

Issue 2: filling the blanks is written twice

Repro
unzip -p dist/skills/integration-session-replay.zip SKILL.md | grep -n '{framework_'
# 3:description: Make session replay record for {framework_name}
# 9:# Make session replay record — {framework_name}

probe2.ts, run with tsx in a wizard#1568 checkout: for each menu entry with frameworkReferences and each integration-* zip that has a variant.json listing that product, it calls the wizard's composeStepSkill(stepFiles, packFiles, product) and this repo's composeStepSkill({ stepFiles, packFiles, product }) and compares the outputs.

steps 6, packs 40, pairs compared 240, outputs differing 0

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its a good callout. we need to rethink how skill delivery to the MCP server and plugins work. most of these skills are only relevant to the wizard harness and its task structure too

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.

shared_docs:
- https://posthog.com/docs/session-replay/installation.md
replaces: replay-vision-enable-replay
framework_references: integration

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, this template doesn't contain {references}, so I think the framework docs and rules get dropped silently. The other five step templates (capture, error-tracking, feature-flags, init, install) all have it, and the last line here still tells the agent to read "the framework docs below".

Adding ## Reference and {references} at the end of description.md should fix it.

type: docs-only
template: ../../replay-vision-enable-replay/description.md
template: description.md
description: Make session replay record for {display_name}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will show {framework_name} as-is in skill-menu.json, because:

  • framework_references is set, so expandSkillGroups replaces {display_name} with {framework_name} (skill-generator.js:447)
  • {framework_name} is only filled in by composeStepSkill when a pack is installed
  • menu, cliEntries, and manifest.json entries never go through composeStepSkill
Suggested change
description: Make session replay record for {display_name}
description: Make session replay record on server and client


- From the project state / MCP tools, note: is session replay recording on, does the project have recordings, and what scanners already exist (`vision-scanners-list` via the PostHog `exec` tool).
- From the repo: is PostHog integrated (`posthog-js` / a server SDK in the dependency manifests, a snippet, or events in the project state)? **If it is not, do not stop** - install and initialize the SDK first, following the `integration-v2-install` and `integration-v2-init` skills (install each with `install_skill`). The minimal SDK footprint they leave behind is enough for recordings to flow; skip identify, capture, and dashboards - this run sets up Replay vision, not the full integration.
- From the repo: is PostHog integrated (`posthog-js` / a server SDK in the dependency manifests, a snippet, or events in the project state)? **If it is not, do not stop** - install and initialize the SDK first, following the `integration-install` and `integration-init` skills (install each with `install_skill`). The minimal SDK footprint they leave behind is enough for recordings to flow; skip identify, capture, and dashboards - this run sets up Replay vision, not the full integration.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it's possible for a caller without an integration pack (for example wizard skill replay-vision-setup or an external MCP client) to follow this and hit no framework pack, or see raw {framework_docs} placeholders. It may not happen today, but maybe guard against it, for example by telling non-orchestrator callers to load the integration family and install the matching framework pack first.

expect(group.variants).toEqual(replaced.variants);
it.each(steps)('%s keeps its template in its own folder', (key, group) => {
expect(group.template).toBe('description.md');
expect(fs.existsSync(path.join(REPO_CONFIG_DIR, 'skills', ...key.split('/'), group.template))).toBe(true);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this suite now only checks that each template exists, so the missing {references} and leaked {framework_name} above pass CI. Maybe assert {references} in each framework_references template, forbid {display_name} in its description, and compose one real step against a pack.

@edwinyjlim
edwinyjlim marked this pull request as ready for review November 17, 2026 17:08
@edwinyjlim
edwinyjlim requested review from a team as code owners November 17, 2026 17:08
@edwinyjlim
edwinyjlim requested review from TueHaulund, fasyy612 and ksvat and removed request for a team November 17, 2026 17:08
edwinyjlim and others added 4 commits November 17, 2026 11:19
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…it a plain description

The template named framework docs below it and listed none, so a completed
skill carried the framework's pages without naming them. Its description
held {display_name}, which the menu showed as a placeholder.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@edwinyjlim
edwinyjlim merged commit 333072f into main Nov 17, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants