MCP: reach every editor feature (settings, captions, regions, clips, media, export) #1180

Decision

The MCP must be able to do everything the editor can: an agent connected over MCP discovers and acts on every project feature, the look included (frames, 3D, webcam cutout, cursor, captions and translations), not only the timeline. App-level secrets (LLM keys, MCP token) stay out.

Follows #1174 (one server, two transports), whose PR 3 brings record and export to the in-app server.

Audited at origin/main @ b4b2890. Paths relative to the repo root. R = readable by MCP, W = tool that writes it.

Exposed today:

• Document tools (electron/ai-edition/deep-agent/service.ts:393-421), on both servers: getCurrentDocument, getTranscript, getTranscriptWords, getCursorTrack, setWordText, removeWords, addTrim(s), setTrim, setClipRange, moveClip, replaceTimeline, addZoom(s), setZoom, addSpeed, setSpeed, addAnnotation, setAnnotation, addCameraFullscreen, setCameraFullscreen, addAudio, setAudio, removeTrim, removeModifier, removeClip, setAspectRatio.
• Both servers also add listProjects, createCheckpoint, restoreCheckpoint.
• Stdio only (electron/mcp/media-tools.ts): listSources, startRecording, stopRecording, setPointerTrack, exportVideo, getFrames. PR 3 of MCP: one server, two transports — the app is the authority when it runs #1174 brings these to the in-app server.

Headline

• The MCP edits the timeline only: clips, trims, zooms, speed, text annotations, camera-fullscreen regions, audio tracks, transcript words, aspect ratio. Almost all of the project's LOOK is unreachable.
• Every legacyEditor setting is invisible to getCurrentDocument except autoFocusAll, and none is writable except aspectRatio. That covers wallpaper, padding, roundness, shadow, blur, frame, motion blur, depth of field, the whole webcam layout, cursor, master gain, format fill, captions and translations. The snapshot omits even aspectRatio.
• Captions: nothing is readable or writable, although service.ts:113 (also the MCP instructions) and editor.json:270,275 promise "add captions". editor.json:276 promises "split a clip", which exists nowhere.
• The CLI openscreen captions (src/cli/CliCaptionsRunner.tsx) writes legacy v1 auto-caption annotations. The v2 CaptionsPane treats those as legacy and offers to clear them, so the CLI is obsolete as well.

1. Background, frame, composition (legacyEditor, typed by src/lib/ai-edition/store/editorSettings.ts:88-134)

2. 3D and camera

3. Webcam (LayoutPane, RightPanes.tsx:2804-3141)

4. Cursor (CursorPane, RightPanes.tsx:3680-3996)

cursorShow, cursorAutoHide, cursorTheme (CURSOR_THEME_IDS, lib/cursor/cursorThemes.ts:290), cursorAsArrow (CURSOR_KIND_IDS, :119), cursorSize [1.5,6], cursorSmoothing, cursorMotionBlur, cursorClickBounce [0,2], cursorClickImpact, cursorClickSound, cursorClickSoundGainDb: all R no, W no. The pointer track is covered by getCursorTrack and setPointerTrack. Check the cursor style modes as well (e.g. Glass Lens).

5. Timeline

6. Annotations (FloatingInspector.tsx:962-1266)

7. Audio

• Master gain audioGainDb ±12 dB: no / no.
• Track fade in and out audioTracks[].fadeInMs/fadeOutMs (max 5000): no / no.
• Import audio or record a voiceover (DocumentService.addAsset, document-service.ts:533): no.

8. Captions (legacyEditor.captions, src/lib/ai-edition/captions/settings.ts:50-127; CaptionsPane.tsx)

• Nothing here is readable or writable:
  • enabled;
  • style (CAPTION_STYLES, settings.ts:182);
  • font, weight, size, colour;
  • plate and opacity (captionPlatePatch, :152);
  • anchors and insets;
  • words per line;
  • lane (resolveCaptionLane, :638);
  • display language and translation layers (legacyEditor.captionTranslations, translations.ts).
• Translation exists only through the app LLM (electron/ai-edition/caption-translate.ts, IPC captions.translate). The MCP client can translate itself and store the result with putCaptionTranslation (translations.ts:85).

9. Transcription

• Transcription is only triggered in the renderer when the editor opens: transcriptionStore.ts, document/transcribe.ts:43, v4/MediaStage.tsx:463. A stdio project from stopRecording has NO transcript until the user opens it, so getTranscript and removeWords fail.
• No tool triggers it, redoes it or forces a language.
• The project vocabulary (project.vocabulary, max 600 chars) and assets[].transcriptionFailure are neither readable nor writable.

10. Export (dialog state)

• exportVideo passes null for GIF fps and size (media-tools.ts:508-509) although the CLI supports them.
• It has no mp4 fps (60 is fixed in lib/cliContracts.ts) and no GIF loop.
• autoZoom and the voiceover mix are hardcoded off (media-tools.ts:510-513).

11. Projects and media

• Rename the project (project.title): no tool.
• Import or remove media (DocumentService.addAsset :533, removeAsset :587): no tool.
• Create a project (:464): only through stopRecording.
• Delete a project: none. It is destructive; leave it out.

12. Recording

• startRecording cannot record the webcam (CliRecordRequest has no camera fields).
• The cursor mode is forced to editable-overlay (media-tools.ts:345).

13. App-level (outside the project)

• Style presets (electron/ai-edition/style-preset-service.ts:115-215; applied with stylePresetPatch, src/lib/ai-edition/stylePresetsEditor.ts:90): list, apply, save, and the preset for new projects.
• LLM keys, MCP token and port, locale, theme, shortcuts: these stay out of the MCP.

Promises no tool keeps

• service.ts:113: "add captions". It is also the MCP instructions (openscreen-mcp-server.ts:230).
• editor.json:270,275,276: "add captions", "split a clip".
• agent-tools.ts:2630: "to place footage use replaceTimeline", but that uses the primary asset only.
• The snapshot reports focusMode, and setters accept annotation style and position, yet nothing reads those back or sets focusMode.

Proposed batches

• A. setEditorSettings + a settings read block in getCurrentDocument.
  • A partial patch named after EditorSettingsPatch (store/editorSettings.ts:343), applied with applySettingsPatch (store/settingsPatch.ts:15) like setAspectRatio (agent-tools.ts:1478).
  • Zod bounds from SETTING_BOUNDS (src/lib/projectDefaults.ts:295) and the enum lists. Wallpapers go through parseStylePresetWallpaper: that is the security boundary, no file:// and no URLs.
  • Read side: getEditorSettings(doc) (editorSettings.ts:213), with data-URL wallpapers elided, plus the gating context (formatFillAvailability, tilted zoom present, bundled wallpapers, segmentation capability if reachable).
  • Optional: listStylePresets and applyStylePreset.
  • Covers §1, the webcam part of §3, §4, autoFocusAll and audioGainDb.
• B. Captions and transcription.
  • setCaptionSettings through patchCaptionSettings (captions/settings.ts:657), with a style shortcut.
  • Translations: read units with captionTranslationUnits / untranslatedUnits, write with putCaptionTranslation, remove with removeCaptionTranslation; the client translates itself.
  • transcribe(assetId, language?) against the main-process STT. When the app runs, use or wait for its transcription.
  • Project vocabulary.
  • Align the CLI captions command on the v2 model, and fix the prompts.
• C. Fill in the region tools.
  • setZoom: focusMode, hideCursor, customScale.
  • setAudio: fades.
  • add/setAnnotation: type, figureData, blurData.
  • Snapshot: annotation style, position and type data.
• D. Clip structure.
  • New tools: insertClip, duplicateClip, addAutoZooms.
  • setClipRange gains cropRegion.
  • Fix the replaceTimeline hint and the "split a clip" promise.
• E. Projects and media: setProjectInfo (title), importMedia, removeMedia, createProject.
• F. Export and record:
  • exportVideo gains gifFrameRate, gifSizePreset, autoZoom, mp4 fps and GIF loop;
  • startRecording gains the webcam.

Plan

One PR per batch A to F above, each adding tools to both transports and keeping instructions, tool descriptions, technical-documentation/architecture/mcp-server.md, website/docs/ai-editing.md (+ translations) and the openscreen-demo skill true.