Repository navigation
feat(orchestrator): read the step skills by name and complete them from the pack - #1566
Conversation
…om the pack The bundled prompts name the integration-* step skills. A skill whose menu entry has frameworkReferences is installed once a run and completed from the framework's pack. When it cannot be completed, the tasks that name it fail as they do when a skill fails to install, and a run that installed no pack fails before any task starts. The wizard-agent-dag-skills flag, the replaces mapping and the fallback to a replaced skill are removed. A context-mill pin older than 1.66.0 is refused, since that is the first release that publishes these skills. The skill menu and the step and pack pair are recorded from context-mill 1.66.0, and record:context-mill-fixtures records the pair. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
install_skill completes a skill whose menu entry has frameworkReferences from the framework's pack, through the installer the orchestrator uses, with the same path and scan checks. The pack is picked with the run's match keys and read from a scratch folder. A run with no framework gets an error that says so, and nothing is installed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A linear error tracking run installs integration-install and integration-init when PostHog is not integrated yet. install_skill completes both from the framework's pack. Comments describe the step skills and the bundled prompts as they are. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
johncwaters
left a comment
There was a problem hiding this comment.
Verdict: REQUEST CHANGES
See inline comments.
Automated review by Glimmervoid. Not written by a human
| // A folder an earlier run left behind would pass its old pages off as the skill's own. | ||
| removeSkillFolder(skillsDir, replacement.id, run.installDir); | ||
| const result = await run.install(replacement.id); | ||
| removeSkillFolder(skillsDir, skill.id, run.installDir); |
There was a problem hiding this comment.
[convergent: boundary + redteam] HIGH
install_skill now comes through here with skillsRoot set to .claude/skills, and removeSkillFolder is a plain rmSync(..., { recursive: true, force: true }) with no check for the .posthog-wizard marker. So a project that already has its own .claude/skills/integration-install (hand-installed, or left by a crashed run) loses it, and if completing fails afterwards nothing replaces it. Before this, install_skill went through downloadSkill, which deleted nothing. Could the tool path delete only folders that carry the marker and refuse an unmarked one?
Automated review. Not written by a human
There was a problem hiding this comment.
Ran it to be sure. In a temp project with my own .claude/skills/integration-install/SKILL.md and no .posthog-wizard marker, I called repackagedSkillInstaller from #1568's head with only the download stubbed to fail, as it would offline:
before: [ 'integration-install' ] marker: false
install rejected: download-failed
after: [] my SKILL.md exists: false
My folder is deleted before the download starts, so a successful install overwrites it too. The fix I'd reach for is to never delete before you have a replacement: install and complete into a wizard temp dir, then swap it in only over a folder that carries the marker, and refuse an unmarked one. That also covers the throw before the try, since a half-finished install never lands in .claude/skills.
Probe
const install = repackagedSkillInstaller({
installDir: project,
skillsRoot: '.claude/skills',
referencePath: 'pack',
install: async () => ({ kind: 'download-failed', message: 'offline' }),
scanFile: async () => null,
onUnavailable: () => {},
});
await install({ id: 'integration-install', frameworkReferences: 'nextjs' } as never);There was a problem hiding this comment.
Fixed in 9d0990e. The skill is now installed, completed and scanned in a temp folder, and only then put in place. It replaces a folder only when it is absent or carries the .posthog-wizard marker, and refuses an unmarked one before anything is downloaded. The orchestrator's own cache is the one root it always replaces. Your repro is a test now: an unmarked folder survives both a failed and a successful install.
| const result = await run.install(replacement.id); | ||
| removeSkillFolder(skillsDir, skill.id, run.installDir); | ||
| const result = await run.install(skill.id); | ||
| if (result.kind !== 'ok') throw new Error(result.kind); |
There was a problem hiding this comment.
[redteam] MEDIUM
This throw sits before the try, so the cleanup below never runs when the install fails. downloadSkill can fail after extracting: a scan engine that throws lands in its catch, which returns success: false and leaves skillDir behind. On the install_skill path that leaves an unscanned step skill, placeholders and all, in .claude/skills/<id>. Moving the install inside the try covers it.
Automated review. Not written by a human
There was a problem hiding this comment.
Fixed in 9d0990e. Nothing is written under .claude/skills until the skill is complete and scanned, and the temp folder is removed in a finally. downloadSkill also removes a folder it created when its install fails.
|
|
||
| expect(compareSkillFiles(original, composed)).toEqual([]); | ||
| expect(composed['SKILL.md']).not.toBe(original['SKILL.md']); | ||
| expect(composed).toEqual(tree(path.join(root, 'completed'))); |
There was a problem hiding this comment.
[code] MEDIUM
The recorder writes completed/ with composeStepSkill (record-context-mill-fixtures.no-jest.ts:103), so this compares composeStepSkill with its own output. If the wizard's fill drifts from context-mill's, recording again regenerates the expected tree and the test stays green. Recording context-mill's published variant as completed/ and comparing with the name line excluded would bring the cross-check back.
Automated review. Not written by a human
There was a problem hiding this comment.
Fixed in 7725d2d. completed/ is now the skill context-mill 1.66.0 publishes for the same step and framework, and the recorder never writes it from the wizard's own fill. The test compares everything but the frontmatter name line.
sarahxsanders
left a comment
There was a problem hiding this comment.
Two issues from the local review; details inline.
from Sarah's review bot
| // A folder an earlier run left behind would pass its old pages off as the skill's own. | ||
| removeSkillFolder(skillsDir, replacement.id, run.installDir); | ||
| const result = await run.install(replacement.id); | ||
| removeSkillFolder(skillsDir, skill.id, run.installDir); |
There was a problem hiding this comment.
Can we preserve unmarked existing skill folders here? installToolSkill now uses this against .claude/skills, so this recursively deletes user-authored files before the download has succeeded. I reproduced this with an unmarked integration-install/my-notes.md: a failed download permanently removes it. Could we stage the completed skill separately and preserve folders we don't own?
from Sarah's review bot
There was a problem hiding this comment.
Fixed in 9d0990e. The completed skill is staged in a temp folder, and a folder without the wizard's marker is refused and left untouched. A test covers an unmarked folder with a failed download.
| removeSkillFolder(skillsDir, replacement.id, run.installDir); | ||
| const result = await run.install(replacement.id); | ||
| removeSkillFolder(skillsDir, skill.id, run.installDir); | ||
| const result = await run.install(skill.id); |
There was a problem hiding this comment.
Can we include the install in the cleanup block? If extraction or the initial scan throws after writing files, installSkillById returns download-failed and we throw before reaching the cleanup below. I reproduced this with an installer that writes SKILL.md and then returns a failure: the unfinished folder stays on disk. On the tool path, that leaves incomplete or unscanned instructions in .claude/skills.
from Sarah's review bot
There was a problem hiding this comment.
Fixed in 9d0990e. The install now happens in the temp folder inside the try, so a failed or unscanned install never reaches .claude/skills, and the temp folder is always removed.
gewenyu99
left a comment
There was a problem hiding this comment.
🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.
| // A folder an earlier run left behind would pass its old pages off as the skill's own. | ||
| removeSkillFolder(skillsDir, replacement.id, run.installDir); | ||
| const result = await run.install(replacement.id); | ||
| removeSkillFolder(skillsDir, skill.id, run.installDir); |
There was a problem hiding this comment.
I think this deletes the existing skill folder even when the download then fails, because:
removeSkillFolderruns beforerun.install- if
installreturns anything butok, we throw, and the old folder is already gone new Error(result.kind)also drops the detail adownload-failedresult carries
Suggestion: install into a temp folder first and only swap it over the old one once install returns ok, and keep the failure detail in the thrown error.
There was a problem hiding this comment.
Fixed in 9d0990e. It stages in a temp folder and only swaps in a finished skill, and the thrown error keeps the detail as download-failed: <message>.
| ...picked, | ||
| frameworkReferences, | ||
| }).catch((error: unknown) => { | ||
| throw new Error( |
There was a problem hiding this comment.
I think if installCompletedSkill fails here (a YARA block or a missing variant), the task gets recorded as no-report instead of the skill error. My line of thought:
- this rethrows, so the executor treats it as a task failure and retries
- if the rejected install is cached, the retry gets the same rejection straight back without the original message reaching the report
- the task then ends with no report, so the real reason is lost
I haven't reproduced this, it's a hunch from reading the code. Worth checking whether the cached rejection surfaces its error.
There was a problem hiding this comment.
Confirmed, and fixed in e880ead. A task whose setup throws is still failed as no-report, but the failure message now carries the thrown reason. The type and the retries are unchanged.
| ? pickSkillId(FRAMEWORK_PACK_FAMILY) | ||
| : undefined; | ||
| if (referenceSkillId) { | ||
| const ref = await installSkillById( |
There was a problem hiding this comment.
I think it'd be good to save the menu somewhere instead of fetching it again here. menuSkillEntries already loaded it, and each installSkillById call fetches skill-menu.json again, which adds up to 22 fetches in one integration run. Maybe fetch it once and resolve upstream.
There was a problem hiding this comment.
fsho, will optimize in a followup
| skillPaths.push(path.join(completed, 'SKILL.md')); | ||
| continue; | ||
| } | ||
| const result = await installSkillById( |
There was a problem hiding this comment.
I think shared plain skills like posthog-best-practices and integration-mcp get downloaded, unzipped and YARA-scanned again for every task that lists them, and parallel tasks can write into the same folder at the same time. Maybe install each one once (memoize by skill id) and reuse the path.
There was a problem hiding this comment.
True, and it predates this PR: main installs shared skills once per task as well. Leaving it for the follow-up with the menu caching.
| skillId: string, | ||
| run: ToolSkillInstall, | ||
| ): Promise<ToolSkillResult> { | ||
| const menu = await fetchSkillMenu(run.skillsBaseUrl); |
There was a problem hiding this comment.
I think each install_skill call fetching the menu and re-downloading the framework pack is a lot of repeated work. Also each call builds its own installer map, so parallel calls don't serialize their writes into .claude/skills. Maybe fetch the menu and pack once per run and share one installer.
There was a problem hiding this comment.
Agreed on the repeated work. Leaving it for the same follow-up as the menu caching. The write overlap is smaller after 9d0990e, since each call builds in its own temp folder and replaces the target whole.
| ); | ||
|
|
||
| // The pack is read once and thrown away, so it never lands in the project. | ||
| const scratch = fs.mkdtempSync(path.join(os.tmpdir(), 'posthog-wizard-')); |
There was a problem hiding this comment.
I think these should be caught. fs.mkdtempSync and fs.rmSync are Node's built-in file functions, and both throw instead of returning an error:
mkdtempSyncthrows when the OS temp folder can't be written, for example a full disk (ENOSPC) or no permission (EACCES)rmSyncin thefinallycan throw when a file is still locked, which happens mostly on Windows with antivirus or an open handle (EBUSY,EPERM)
In both cases installToolSkill rejects instead of returning incomplete. Wrapping both in try/catch (and ignoring cleanup errors in finally) would keep the tool result consistent.
Also, deleting the null-menu or failed-packInstall guards still keeps the tool skill tests green, so those paths could use a test.
There was a problem hiding this comment.
Fixed in 0c656ce. The scratch folder is made inside the try and cleanup errors are swallowed, so the call returns incomplete. Tests added for the null menu and the failed pack install.
| ) as { product: string }; | ||
| const original = tree(path.join(root, 'original')); | ||
|
|
||
| const composed = composeStepSkill( |
There was a problem hiding this comment.
test nit: the completed/ fixtures are recorded with wizard's own composeStepSkill, so if it drifts from context-mill's builder, re-recording overwrites the expected output and this test still passes. Maybe record completed/ from context-mill's build output instead.
There was a problem hiding this comment.
Fixed in 7725d2d. completed/ is recorded from the skill context-mill 1.66.0 publishes, never from the wizard's own fill.
| import { | ||
| incompleteSkillMessage, | ||
| installToolSkill, | ||
| } from '../runner/sequence/orchestrator/repackaged-skills'; |
There was a problem hiding this comment.
nit: importing installToolSkill from the orchestrator means linear install_skill calls send orchestrator repackaged skill unavailable analytics and orchestrator log tags. Maybe pass the event name/tag in, or move the shared part out of orchestrator/.
There was a problem hiding this comment.
Fair. Not moved in this PR. The event keeps its name because the rollout dashboard reads it. Moving the installer out of orchestrator/ can go with the follow-up.
| `Invalid context-mill version "${value}". Use a release version such as ${CONTEXT_MILL_FLOOR}, or "latest".`, | ||
| ); | ||
| } | ||
| if (belowFloor([Number(match[2]), Number(match[3]), Number(match[4])])) { |
There was a problem hiding this comment.
nit: 1.66.0-rc.1 slips past the floor check, since belowFloor compares only major.minor.patch and ignores the prerelease suffix. Also CONTEXT_MILL_FLOOR is exported but has no callers outside this file.
There was a problem hiding this comment.
Fixed in 70738e2. A prerelease of 1.66.0 is refused, and CONTEXT_MILL_FLOOR is no longer exported.
|
✅ Security review complete — 1 finding posted as a review, in 7 min. Add the |
…s the wizard does not own A step skill is installed, completed and scanned in a temp folder outside the project, and put in place only when it is complete. In .claude/skills it replaces a folder only when that folder carries the wizard's marker; one the wizard did not install, or a link standing there, is refused before anything is downloaded. A failed install keeps its detail, and a download that fails removes the folder it made and keeps one that was there first. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A menu entry's id names the folder an install writes to. downloadSkill refuses one that is not a single plain name before it makes a folder or fetches anything, so no menu can place a skill outside the skills folder. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ublishes completed/ is the skill context-mill 1.66.0 itself publishes for the pair's step and framework, read from the skill family the step's menu entry replaces. The recorder never writes it from the wizard's own fill, and leaves it as it is when a release no longer publishes that family. The test compares the two with only the frontmatter name line left out. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A task whose runner throws is failed as no-report once its attempts are spent. Its failure message now carries the thrown error, so a skill that could not be installed reads as that and not as an agent that never reported. The type and the retry behaviour are unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The scratch folder a framework pack is read from is made inside the try, so a folder that cannot be made is reported as a skill that could not be completed, and one that cannot be removed no longer replaces the result of the install. A pack that fails to install keeps the detail of its failure. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A prerelease of 1.66.0 comes before 1.66.0, so a pin to one is refused like any older release. The floor is used only in its own file and is no longer exported. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A completed skill is built in a temp folder and replaces its folder whole, so nothing calls removeSkillFolder. skillFolder keeps its checks: one plain id, and a skills root reached through no link. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
✅ Security review complete — No findings above the confidence threshold, in 14 min. Add the |
sarahxsanders
left a comment
There was a problem hiding this comment.
looks like my bot's findings are fixed!!!
gewenyu99
left a comment
There was a problem hiding this comment.
The fixes from the first round look good to me. Three non-blocking comments inline, none of them need to hold this PR. Thanks!
| return { success: true }; | ||
| } catch (err: any) { | ||
| logToFile(`downloadSkill: error: ${err.message}`); | ||
| if (!existed) { |
There was a problem hiding this comment.
I think this cleanup can delete a skill another task just installed, because:
- parallel tasks install the same plain skill, such as
posthog-best-practices, into the same cache folder - the first call sees
existed === false, the second seestrue - if the first download fails after the second succeeds, the first removes the folder and the second task gets a
SKILL.mdpath that no longer exists
I reproduced it with two concurrent downloadSkill calls where the first gets a late 404: the second returns success: true and SKILL.md is gone. It needs one of two same-URL downloads to fail, so it is rare. Installing each plain skill once per run, the follow-up you already planned, removes it.
| // Only the menu is replaced; `repackaged/` belongs to `verify:repackaged-skills --record`. | ||
| fs.mkdirSync(OUT_DIR, { recursive: true }); | ||
| fs.writeFileSync(path.join(OUT_DIR, 'skill-menu.json'), skillMenu); | ||
| // A folder that is not recorded again stays as it is. |
There was a problem hiding this comment.
nit: once context-mill#537 ships, a release has no integration-v2-* skill and no replaces, so the recorder keeps completed/ from 1.66.0 and still records step/ and pack/ from the new release. The fixture test then fails as soon as the Flask pack or the init template changes, with no way to record a matching completed/. Maybe leave the whole pair alone when the release publishes no skill for it.
| const target = landingFolder(skill, run); | ||
| fs.rmSync(target, { recursive: true, force: true }); | ||
| fs.mkdirSync(path.dirname(target), { recursive: true }); | ||
| fs.cpSync(staged, target, { recursive: true }); |
There was a problem hiding this comment.
nit: if cpSync fails partway, for example on a full disk, the old folder is already removed and a partial one may be left in .claude/skills without the marker. Every later install_skill for that id then refuses it as not the wizard's. Maybe remove target when the copy throws.

Behaviour
A task reads the step skills by their own names, and a step skill written once for every framework is always completed from the framework's pack.
integration-*step skills, such asintegration-installandintegration-session-replay.frameworkReferencesis installed once a run into an emptied folder, completed from the framework's pack, and its rewrittenSKILL.mdis scanned.orchestrator repackaged skill unavailableonce, with the skill, the reason and the framework.install_skillcompletes such a skill the same way, in both sequences and both harnesses, through the same installer. It picks the pack with the run's match keys and reads it from a scratch folder. A run with no framework gets an error that says so, and nothing is installed.--context-mill-versionrefuses a release older than 1.66.0, the first that publishes these skills. The message names that release.The
wizard-agent-dag-skillsflag, thereplacesmapping, the fallback to a replaced skill, theorchestrator_skillstag andpnpm verify:repackaged-skillsare gone.Fixtures
The skill menu and the step and pack pair are recorded from context-mill 1.66.0.
pnpm record:context-mill-fixturesnow records the pair too: the step skill as published, the pack files that completing it reads, and the completed skill. The recorded completed skill differs from the skill context-mill 1.66.0 publishes for the same step and framework in one line, thenamein its frontmatter.Verified
pnpm typecheckandpnpm lint(0 errors).pnpm test: 292 files, 4,450 tests pass, 1 skipped.repackaged-skills.test.tsandrepackaged-skill-installer.test.tscover each way a completion fails: a failed install, a pack that cannot complete the skill, a scan that rejects or fails, a link on the way, and a run with no pack.tool-skill-install.test.tscoversinstall_skill: a plain skill, a completed one, a run with no framework, and a rejected scan. The pack it reads is gone from disk afterwards.context-mill-version.test.tscovers the floor.Changes in behaviour to know about
integration-installandintegration-init, from the menu'sintegration/installandintegration/initcategories.install_skillcompletes both from the framework's pack.install_skilltakes an exact menu id, as before. A name that is a skill family, such asintegration-v2-install, is not found.FEATURE_FLAGS_UNSUPPORTEDstill stops feature flags on Kotlin Multiplatform.Rollout
This is the second of four stacked PRs that remove the old code paths. They merge together and ship in one release.
wizard-agent-dagandwizard-agent-dag-skillshave both held at 100% with no stop rule tripped, no run that fetched its prompts, and no run that fell back to an old skill.🤖 Generated with Claude Code