Skip to content

feat(menu): say which projects a skill serves in its tags alone - #538

Merged
edwinyjlim merged 7 commits into
dag/m2-delete-old-step-skillsfrom
dag/m3-delete-old-menu-fields
Nov 17, 2026
Merged

edwinyjlim merged 7 commits into
dag/m2-delete-old-step-skillsfrom
dag/m3-delete-old-menu-fields

Conversation

@edwinyjlim

@edwinyjlim edwinyjlim commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

What this does

A skill's menu entry says which projects it serves in its tags, and nowhere else.

{
  "id": "integration-nextjs-app-router",
  "name": "PostHog integration for Next.js App Router applications",
  "downloadUrl": "…/integration-nextjs-app-router.zip",
  "title": "Next.js App Router",
  "description": "PostHog integration for Next.js App Router applications",
  "tags": ["nextjs", "react", "…", "framework:nextjs", "frameworkVariant:app-router", "integration", "default"]
}
  • An entry has id, name, downloadUrl, title, description and tags. A step skill also has frameworkReferences.
  • The framework a skill serves is its framework:<id> tag, and its default mark is the default tag. Its group is the category it is listed under in categories, with / read as -. The tags and the categories are the ones a release carries today.
  • An entry has no group, framework or default field.
  • cliEntries is unchanged, including the default of a command family.

Every skill publishes as its own zip:

  • A group cannot be packaged as one JSON of its variants. packaging: bundle, the bundle writer, the dev server's /skills/<group>.json route and the group JSON upload step in build-release.yml are gone.
  • A group cannot borrow another group's variants. variants_from is gone. No config sets either key, and a config that sets one fails the build with a message that names the key.
  • The old-wizard compatibility test and its snapshot are gone, since they check resolution by the removed fields.

Nothing inside context-mill reads an entry's group, framework or default: the manifest, the marketplace generator, the mirror and the dev server work from the skill list or from downloadUrl.

Verification

  • npm test: 368 tests pass.
  • The build passes, and scripts/mirror-dist.js runs on its output.
  • A build of this branch was compared with a build of the branch below it, both from a clean checkout. Only skill-menu.json differs: across its 254 entries, group is gone from 254, framework from 108 and default from 31. Every entry's tags are the same, and so is cliEntries. No skill zip, the manifest, the MCP archive or the marketplace tree differs.

Rollout

Stacked on the step skill removal.

A wizard that picks a skill by an entry's group, framework and default fields finds none once this ships. Merge it only when all of these hold:

  • cloud runs a wizard release that reads a skill's group from its category, and its framework and default from its tags
  • older wizard versions have drained from the run data
  • wizard-agent-dag and wizard-agent-dag-skills are both still on
  • the marketplace and MCP server owners have agreed

🤖 Generated with Claude Code

A skill's menu entry carries `id`, `name`, `downloadUrl`, `title`,
`description` and `tags`, and `frameworkReferences` where a step skill sets
it. The framework a skill serves, its group and its `default` mark are tags.
The entry has no `group`, `framework` or `default` field.

Every skill publishes as its own zip. A group cannot be packaged as one
JSON of its variants, and cannot borrow another group's variants.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@edwinyjlim
edwinyjlim added this pull request to stack #539 November 17, 2026 01:10
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci feature-flags
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci feature-flags/django
  • /wizard-ci feature-flags/next-js
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/go-sdk
  • /wizard-ci mcp-analytics/python-sdk
  • /wizard-ci mcp-analytics/ruby-sdk
  • /wizard-ci mcp-analytics/rust-sdk
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a wizard branch:

  • /wizard-ci all wizard:my-branch

Add wizard:<branch> to any command above to pin the wizard branch. It defaults to main.

Results will be posted here when complete.

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.

Comment thread scripts/dev-server.js Outdated
Comment on lines 260 to 265
const skillMatch = req.url?.match(/^\/skills\/(.+\.zip)$/);
if (skillMatch) {
const [skillFile, ext] = [skillMatch[1], skillMatch[2]];
const isZip = ext === 'zip';
serveFile(
res,
path.join(skillsDir, skillFile),
isZip ? 'application/zip' : 'application/json',
isZip ? skillFile : undefined,
);
const skillFile = skillMatch[1];
serveFile(res, path.join(skillsDir, skillFile), 'application/zip', skillFile);
return;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a path bug here. The regex accepts any .+\.zip, and Node doesn't normalize req.url, so /skills/../something.zip gets joined straight onto skillsDir and can read outside it. A directory whose name ends in .zip also passes existsSync and then crashes the stream with EISDIR.

Restricting the name and checking the resolved path should cover both:

Suggested change
const skillMatch = req.url?.match(/^\/skills\/(.+\.zip)$/);
if (skillMatch) {
const [skillFile, ext] = [skillMatch[1], skillMatch[2]];
const isZip = ext === 'zip';
serveFile(
res,
path.join(skillsDir, skillFile),
isZip ? 'application/zip' : 'application/json',
isZip ? skillFile : undefined,
);
const skillFile = skillMatch[1];
serveFile(res, path.join(skillsDir, skillFile), 'application/zip', skillFile);
return;
}
const skillMatch = req.url?.match(/^\/skills\/([a-z0-9][a-z0-9_-]*\.zip)$/);
if (skillMatch) {
const skillFile = skillMatch[1];
const resolved = path.resolve(skillsDir, skillFile);
if (!resolved.startsWith(path.resolve(skillsDir) + path.sep)) {
res.writeHead(400, { 'Content-Type': 'text/plain', ...NO_CACHE_HEADERS });
res.end('Invalid skill path.');
return;
}
serveFile(res, resolved, 'application/zip', path.basename(skillFile));
return;
}

if (fs.existsSync(configFile)) {
const localConfig = loadYaml(configFile);
if (localConfig?.variants || localConfig?.variants_from) {
if (localConfig?.variants) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a config that still uses variants_from (or packaging: bundle) gets skipped silently here, because:

  • this only checks for a truthy variants
  • with no inline variants, the config is never added, and the build exits 0

It'd be safer to throw (or at least warn), the same way variant.framework already does:

Suggested change
if (localConfig?.variants) {
if (localConfig?.variants_from !== undefined || localConfig?.packaging === 'bundle') {
throw new Error(`Unsupported legacy config key in ${configFile}`);
}
if (localConfig?.variants) {

expect(entry).toMatchObject({ title: 'Ruby on Rails', description: 'rails description', framework: 'rails', default: true });
expect(entry.tags).toEqual(['ruby-on-rails', 'framework:rails', 'integration-v2-capture', 'default']);
expect(entry).toMatchObject({ title: 'Ruby on Rails', description: 'rails description' });
expect(entry.tags).toEqual(['ruby-on-rails', 'framework:rails', 'logs', 'default']);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Hey, I think this is missing coverage. Every case sets default: true, so dropping .filter(Boolean) in menuTags would put null into non-default tags and tests stay green. There's also no check on skill-menu.json built from the real context/ directory anymore. Would be useful to test both, just in case this causes bugs in the future.

'cloudflare-workers.md',
'integration-v2.json',
'skill-menu.json',
'skills-mcp-resources.zip',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this only checks that skills-mcp-resources.zip exists, so an empty archive or stale download URLs would still pass. Maybe open it and assert its entries and the rewritten manifest.json URLs.

expect(entry.resource.text).toBe(entry.downloadUrl);
});

it('never emits a file for a resource the build does not ship as a zip', () => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: with bundle: true removed, this repeats the logs-nextjs.zip and inlined doc cases tested right around it. Probably safe to delete.

@edwinyjlim
edwinyjlim marked this pull request as ready for review November 17, 2026 17:08
@edwinyjlim
edwinyjlim requested a review from a team as a code owner November 17, 2026 17:08
@@ -1,130 +0,0 @@
// Wizard versions already in the field resolve the skills their prompts name with the logic vendored below.

@johncwaters johncwaters Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a line fix, more a design question for the whole stack. A skill's identity is still its folder path: skill-generator.js builds the menu id and category from the config key, and the wizard turns the category into the family (skillFamily, so integration/install becomes integration-install). With old-wizard-compat.test.js gone, nothing ties those names to what shipped wizards ask for, so moving a folder is a breaking change that every check passes.

I ran it at this PR's head to make sure. I renamed context/skills/integration/install to integration/setup and updated the step list in step-skills.test.js to match. All 368 tests pass and pnpm build exits 0. Then I ran the wizard's own prepareSkillMenu and resolveSkill from PostHog/wizard#1568 over the built menu: integration-install, which five bundled prompts name, resolves to undefined, so every wizard on this stack fails preflight with "no longer published".

I don't think the fix is another CI test. Could identity be a field the skill declares instead of one we derive?

  • id: and aliases: in config.yaml, so a folder move or a rename keeps the old name resolving
  • match: kept structured in the menu, rather than flattened into key:value tags the wizard parses back out
  • a published skill-menu.schema.json with a schemaVersion, and the wizard's SkillEntry generated from it instead of written by hand

That leaves deleting an id as the only way to break a wizard, and the build can refuse that by diffing ids against the last published menu.

Issue 1: menu contract has no guard

Repro
git archive cb8d3f15 | tar -x -C renamed && cd renamed
mv context/skills/integration/install context/skills/integration/setup
# step-skills.test.js: 'integration-install' -> 'integration-setup'
pnpm test    # Tests  368 passed (368)
pnpm build   # exit 0

probe.ts, run with tsx against prepareSkillMenu and resolveSkill copied verbatim from wizard#1568 (src/shared/skill-menu.ts, src/shared/skill-resolution.ts):

const menu = prepareSkillMenu(JSON.parse(readFileSync(menuPath, 'utf8')));
const entries = Object.values(menu.categories).flat();
resolveSkill(entries, 'integration-install', { framework: ['nextjs'] });
unchanged: resolveSkill('integration-install', nextjs) -> integration-install
renamed:   resolveSkill('integration-install', nextjs) -> undefined

@johncwaters johncwaters Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same root, other direction. CONTEXT_MILL_FLOOR stops a new wizard from reading an old context-mill, but wizards already released read releases/latest/download/skill-menu.json, so they get this menu the day it ships.

I ran wizard v2.95.0's resolveSkill over two builds. On today's main menu, integration-v2-install, integration-v2-init and integration-v2-capture all resolve. On the menu this stack builds, all three come back undefined. That's the lookup v2.95.0 runs with wizard-agent-dag on. With the flag off, it fetches agent-menu.json first, and this stack stops publishing that file. Either way, every v2.95.0 install fails its run once this releases.

The only guard is the merge gate in the PR body, which no code checks. The root cause is one moving URL for every wizard ever released. Could the wizard ask for skill-menu.v<major>.json instead, with context-mill keeping the previous major's files published until our wizard version analytics show nobody's on it? That makes a breaking menu change a new file instead of a broken one.

Issue 3: the version guard faces one way

Repro
# context-mill: build origin/main (8594917e) and this PR's head (cb8d3f15)
find main/dist -name agent-menu.json | wc -l    # 1
find stack/dist -name agent-menu.json | wc -l   # 0

probe3.ts, run with tsx in a wizard checkout at main (v2.95.0 plus one commit), flattening menu.categories and calling resolveSkill(entries, name, { framework: ['nextjs'] }):

main:  v2.95.0 resolveSkill('integration-v2-install', nextjs) -> integration-v2-install
main:  v2.95.0 resolveSkill('integration-v2-init', nextjs) -> integration-v2-init
main:  v2.95.0 resolveSkill('integration-v2-capture', nextjs) -> integration-v2-capture
stack: v2.95.0 resolveSkill('integration-v2-install', nextjs) -> undefined
stack: v2.95.0 resolveSkill('integration-v2-init', nextjs) -> undefined
stack: v2.95.0 resolveSkill('integration-v2-capture', nextjs) -> undefined

edwinyjlim and others added 6 commits November 17, 2026 11:19
A request path was joined onto the skills folder as written, so one with ..
read a zip outside it, and a folder named like a zip crashed the stream.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Such a config lists no variants of its own, so the build skipped it and
passed with the skill missing.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Also drops a manifest test that repeats the two beside it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@edwinyjlim
edwinyjlim merged commit 4a9250f into main Nov 17, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants