Repository navigation
feat(menu): say which projects a skill serves in its tags alone - #538
Conversation
A skill's menu entry carries `id`, `name`, `downloadUrl`, `title`, `description` and `tags`, and `frameworkReferences` where a step skill sets it. The framework a skill serves, its group and its `default` mark are tags. The entry has no `group`, `framework` or `default` field. Every skill publishes as its own zip. A group cannot be packaged as one JSON of its variants, and cannot borrow another group's variants. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a wizard branch:
Add Results will be posted here when complete. |
gewenyu99
left a comment
There was a problem hiding this comment.
🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.
| const skillMatch = req.url?.match(/^\/skills\/(.+\.zip)$/); | ||
| if (skillMatch) { | ||
| const [skillFile, ext] = [skillMatch[1], skillMatch[2]]; | ||
| const isZip = ext === 'zip'; | ||
| serveFile( | ||
| res, | ||
| path.join(skillsDir, skillFile), | ||
| isZip ? 'application/zip' : 'application/json', | ||
| isZip ? skillFile : undefined, | ||
| ); | ||
| const skillFile = skillMatch[1]; | ||
| serveFile(res, path.join(skillsDir, skillFile), 'application/zip', skillFile); | ||
| return; | ||
| } |
There was a problem hiding this comment.
I think there's a path bug here. The regex accepts any .+\.zip, and Node doesn't normalize req.url, so /skills/../something.zip gets joined straight onto skillsDir and can read outside it. A directory whose name ends in .zip also passes existsSync and then crashes the stream with EISDIR.
Restricting the name and checking the resolved path should cover both:
| const skillMatch = req.url?.match(/^\/skills\/(.+\.zip)$/); | |
| if (skillMatch) { | |
| const [skillFile, ext] = [skillMatch[1], skillMatch[2]]; | |
| const isZip = ext === 'zip'; | |
| serveFile( | |
| res, | |
| path.join(skillsDir, skillFile), | |
| isZip ? 'application/zip' : 'application/json', | |
| isZip ? skillFile : undefined, | |
| ); | |
| const skillFile = skillMatch[1]; | |
| serveFile(res, path.join(skillsDir, skillFile), 'application/zip', skillFile); | |
| return; | |
| } | |
| const skillMatch = req.url?.match(/^\/skills\/([a-z0-9][a-z0-9_-]*\.zip)$/); | |
| if (skillMatch) { | |
| const skillFile = skillMatch[1]; | |
| const resolved = path.resolve(skillsDir, skillFile); | |
| if (!resolved.startsWith(path.resolve(skillsDir) + path.sep)) { | |
| res.writeHead(400, { 'Content-Type': 'text/plain', ...NO_CACHE_HEADERS }); | |
| res.end('Invalid skill path.'); | |
| return; | |
| } | |
| serveFile(res, resolved, 'application/zip', path.basename(skillFile)); | |
| return; | |
| } |
| if (fs.existsSync(configFile)) { | ||
| const localConfig = loadYaml(configFile); | ||
| if (localConfig?.variants || localConfig?.variants_from) { | ||
| if (localConfig?.variants) { |
There was a problem hiding this comment.
I think a config that still uses variants_from (or packaging: bundle) gets skipped silently here, because:
- this only checks for a truthy
variants - with no inline
variants, the config is never added, and the build exits 0
It'd be safer to throw (or at least warn), the same way variant.framework already does:
| if (localConfig?.variants) { | |
| if (localConfig?.variants_from !== undefined || localConfig?.packaging === 'bundle') { | |
| throw new Error(`Unsupported legacy config key in ${configFile}`); | |
| } | |
| if (localConfig?.variants) { |
| expect(entry).toMatchObject({ title: 'Ruby on Rails', description: 'rails description', framework: 'rails', default: true }); | ||
| expect(entry.tags).toEqual(['ruby-on-rails', 'framework:rails', 'integration-v2-capture', 'default']); | ||
| expect(entry).toMatchObject({ title: 'Ruby on Rails', description: 'rails description' }); | ||
| expect(entry.tags).toEqual(['ruby-on-rails', 'framework:rails', 'logs', 'default']); |
There was a problem hiding this comment.
nit: Hey, I think this is missing coverage. Every case sets default: true, so dropping .filter(Boolean) in menuTags would put null into non-default tags and tests stay green. There's also no check on skill-menu.json built from the real context/ directory anymore. Would be useful to test both, just in case this causes bugs in the future.
| 'cloudflare-workers.md', | ||
| 'integration-v2.json', | ||
| 'skill-menu.json', | ||
| 'skills-mcp-resources.zip', |
There was a problem hiding this comment.
nit: this only checks that skills-mcp-resources.zip exists, so an empty archive or stale download URLs would still pass. Maybe open it and assert its entries and the rewritten manifest.json URLs.
| expect(entry.resource.text).toBe(entry.downloadUrl); | ||
| }); | ||
|
|
||
| it('never emits a file for a resource the build does not ship as a zip', () => { |
There was a problem hiding this comment.
nit: with bundle: true removed, this repeats the logs-nextjs.zip and inlined doc cases tested right around it. Probably safe to delete.
| @@ -1,130 +0,0 @@ | |||
| // Wizard versions already in the field resolve the skills their prompts name with the logic vendored below. | |||
There was a problem hiding this comment.
Not a line fix, more a design question for the whole stack. A skill's identity is still its folder path: skill-generator.js builds the menu id and category from the config key, and the wizard turns the category into the family (skillFamily, so integration/install becomes integration-install). With old-wizard-compat.test.js gone, nothing ties those names to what shipped wizards ask for, so moving a folder is a breaking change that every check passes.
I ran it at this PR's head to make sure. I renamed context/skills/integration/install to integration/setup and updated the step list in step-skills.test.js to match. All 368 tests pass and pnpm build exits 0. Then I ran the wizard's own prepareSkillMenu and resolveSkill from PostHog/wizard#1568 over the built menu: integration-install, which five bundled prompts name, resolves to undefined, so every wizard on this stack fails preflight with "no longer published".
I don't think the fix is another CI test. Could identity be a field the skill declares instead of one we derive?
id:andaliases:inconfig.yaml, so a folder move or a rename keeps the old name resolvingmatch:kept structured in the menu, rather than flattened intokey:valuetags the wizard parses back out- a published
skill-menu.schema.jsonwith aschemaVersion, and the wizard'sSkillEntrygenerated from it instead of written by hand
That leaves deleting an id as the only way to break a wizard, and the build can refuse that by diffing ids against the last published menu.
Repro
git archive cb8d3f15 | tar -x -C renamed && cd renamed
mv context/skills/integration/install context/skills/integration/setup
# step-skills.test.js: 'integration-install' -> 'integration-setup'
pnpm test # Tests 368 passed (368)
pnpm build # exit 0probe.ts, run with tsx against prepareSkillMenu and resolveSkill copied verbatim from wizard#1568 (src/shared/skill-menu.ts, src/shared/skill-resolution.ts):
const menu = prepareSkillMenu(JSON.parse(readFileSync(menuPath, 'utf8')));
const entries = Object.values(menu.categories).flat();
resolveSkill(entries, 'integration-install', { framework: ['nextjs'] });unchanged: resolveSkill('integration-install', nextjs) -> integration-install
renamed: resolveSkill('integration-install', nextjs) -> undefined
There was a problem hiding this comment.
Same root, other direction. CONTEXT_MILL_FLOOR stops a new wizard from reading an old context-mill, but wizards already released read releases/latest/download/skill-menu.json, so they get this menu the day it ships.
I ran wizard v2.95.0's resolveSkill over two builds. On today's main menu, integration-v2-install, integration-v2-init and integration-v2-capture all resolve. On the menu this stack builds, all three come back undefined. That's the lookup v2.95.0 runs with wizard-agent-dag on. With the flag off, it fetches agent-menu.json first, and this stack stops publishing that file. Either way, every v2.95.0 install fails its run once this releases.
The only guard is the merge gate in the PR body, which no code checks. The root cause is one moving URL for every wizard ever released. Could the wizard ask for skill-menu.v<major>.json instead, with context-mill keeping the previous major's files published until our wizard version analytics show nobody's on it? That makes a breaking menu change a new file instead of a broken one.
Repro
# context-mill: build origin/main (8594917e) and this PR's head (cb8d3f15)
find main/dist -name agent-menu.json | wc -l # 1
find stack/dist -name agent-menu.json | wc -l # 0probe3.ts, run with tsx in a wizard checkout at main (v2.95.0 plus one commit), flattening menu.categories and calling resolveSkill(entries, name, { framework: ['nextjs'] }):
main: v2.95.0 resolveSkill('integration-v2-install', nextjs) -> integration-v2-install
main: v2.95.0 resolveSkill('integration-v2-init', nextjs) -> integration-v2-init
main: v2.95.0 resolveSkill('integration-v2-capture', nextjs) -> integration-v2-capture
stack: v2.95.0 resolveSkill('integration-v2-install', nextjs) -> undefined
stack: v2.95.0 resolveSkill('integration-v2-init', nextjs) -> undefined
stack: v2.95.0 resolveSkill('integration-v2-capture', nextjs) -> undefined
A request path was joined onto the skills folder as written, so one with .. read a zip outside it, and a folder named like a zip crashed the stream. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Such a config lists no variants of its own, so the build skipped it and passed with the skill missing. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Also drops a manifest test that repeats the two beside it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>


What this does
A skill's menu entry says which projects it serves in its tags, and nowhere else.
{ "id": "integration-nextjs-app-router", "name": "PostHog integration for Next.js App Router applications", "downloadUrl": "…/integration-nextjs-app-router.zip", "title": "Next.js App Router", "description": "PostHog integration for Next.js App Router applications", "tags": ["nextjs", "react", "…", "framework:nextjs", "frameworkVariant:app-router", "integration", "default"] }id,name,downloadUrl,title,descriptionandtags. A step skill also hasframeworkReferences.framework:<id>tag, and its default mark is thedefaulttag. Its group is the category it is listed under incategories, with/read as-. The tags and the categories are the ones a release carries today.group,frameworkordefaultfield.cliEntriesis unchanged, including thedefaultof a command family.Every skill publishes as its own zip:
packaging: bundle, the bundle writer, the dev server's/skills/<group>.jsonroute and the group JSON upload step inbuild-release.ymlare gone.variants_fromis gone. No config sets either key, and a config that sets one fails the build with a message that names the key.Nothing inside context-mill reads an entry's
group,frameworkordefault: the manifest, the marketplace generator, the mirror and the dev server work from the skill list or fromdownloadUrl.Verification
npm test: 368 tests pass.scripts/mirror-dist.jsruns on its output.skill-menu.jsondiffers: across its 254 entries,groupis gone from 254,frameworkfrom 108 anddefaultfrom 31. Every entry'stagsare the same, and so iscliEntries. No skill zip, the manifest, the MCP archive or the marketplace tree differs.Rollout
Stacked on the step skill removal.
A wizard that picks a skill by an entry's
group,frameworkanddefaultfields finds none once this ships. Merge it only when all of these hold:wizard-agent-dagandwizard-agent-dag-skillsare both still on🤖 Generated with Claude Code