Skip to content

feat(agent): region tools set and read every zoom, annotation and audio setting - #1182

Merged
EtienneLescot merged 2 commits into
mainfrom
feat/mcp-region-tools-1180c
Oct 13, 2026
Merged

EtienneLescot merged 2 commits into
mainfrom
feat/mcp-region-tools-1180c

Conversation

@EtienneLescot

@EtienneLescot EtienneLescot commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Batch C of #1180: the region tools reach every setting the inspector has, and getCurrentDocument reads them back. Both transports get it (shared tool table), the in-app chat agent too.

Zooms (addZoom(s), setZoom)

  • focusMode (manual / auto) and hideCursor, written as the inspector writes them.
  • customScale: rounded to the hundredth and stored as a depth when the table has that level (zoomLevelFor, now shared with the inspector's level control). Refused past the clip's zoomScaleLimit, with the limit in the message, as the inspector refuses it. depth with customScale is refused.

Annotations (addAnnotation, setAnnotation)

  • type: text, arrow (figure) or privacy blur, converted through convertAnnotationKind. A retype uses the footage under each fragment (annotationFootageRect), like the type switch.
  • figureData / blurData validated by the schema's figureDataSchema / blurDataSchema (now exported). On setAnnotation only the keys sent change.
  • width / height for arrows and blurs. A blur's box is the recording's picture, so it stays on what it hides.
  • Refused: arguments of another kind (width on a text, blurData on an arrow) and a freehand blur, which the inspector no longer offers. Images are left out and the description says so.

Audio (addAudio, setAudio)

  • fadeInMs / fadeOutMs, capped at AUDIO_FADE_MAX_MS (5000), moved out of RightPanes.tsx so the pane and the tools share it.

Read side

  • Each annotation: type, box (frame / footage), centre x/y, width/height, a text's look, an arrow's figureData, a blur's blurData. An image no longer dumps its data URL as text.
  • Each zoom: hideCursor when set (customScale was already there). Each audio track: its fades.

Product decisions

  • Out-of-range levels and fades are refused, not clamped, so the agent hears the limit.
  • addAudio keeps fades at 0 by default, as before. The editor's music defaults (−18 dB, 1 s fades) are not applied to agent tracks.

Docs: tool descriptions, system prompt, ai-agent.md, website/docs/ai-editing.md and its 7 translations, the openscreen-demo skill. mcp-server.md needed no change: it lists no per-tool arguments.

Related issue

Part of #1180

Type of change

  • Bug fix
  • Feature
  • Enhancement
  • Documentation
  • Refactor / maintenance
  • Performance
  • Security

Release impact

  • Patch
  • Minor
  • Major / breaking change
  • No release note needed

Desktop impact

  • Windows
  • macOS
  • Linux
  • Installer / packaging
  • Not platform-specific

Screenshots / video

None: no UI change. The inspector's level control and audio pane behave as before.

Testing

  • New Vitest cases in electron/ai-edition/agent-tools.test.ts (zoom settings, level limit, blur, arrow, retype, refusals, fades) and deep-agent/service.test.ts (descriptions).
  • npx tsc --noEmit, npx tsc -p tsconfig.test.json --noEmit, npm run lint.
  • npx vitest --run electron/ai-edition electron/mcp src/lib/ai-edition: 1746 passed.
  • Not run: the app itself (no window launched), the website build.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • AI-assisted editing now supports text, arrow, and privacy-blur annotations, with type-specific positioning and settings.
    • Add and adjust zoom focus, cursor visibility, and custom scale, with limits based on the clip.
    • Set audio fade-in and fade-out durations, up to five seconds.
    • Document snapshots now include annotation, zoom, and audio settings.
  • Documentation
    • Updated AI editing guides to describe annotation types, audio fades, and zoom controls.

…io setting

setZoom/addZoom take focusMode, hideCursor and customScale (written as the inspector writes a typed level, refused past the clip's zoomScaleLimit). add/setAnnotation take the type (text, arrow, privacy blur) through convertAnnotationKind, with figureData/blurData validated by the schema's own. add/setAudio take fades capped at the inspector's AUDIO_FADE_MAX_MS. getCurrentDocument reads all of it back.

Part of #1180
@coderabbitai

coderabbitai Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9b51921d-e9aa-40b0-b9fd-c7078c9c5c0b
📥 Commits

Reviewing files that changed from the base of the PR and between 682862d and 9ffcb4e.

📒 Files selected for processing (19)
  • electron/ai-edition/agent-tools.test.ts
  • electron/ai-edition/agent-tools.ts
  • electron/ai-edition/deep-agent/service.test.ts
  • electron/ai-edition/deep-agent/service.ts
  • skills/openscreen-demo/SKILL.md
  • src/components/ai-edition/RightPanes.tsx
  • src/components/ai-edition/v4/FloatingInspector.tsx
  • src/lib/ai-edition/document/audioTracks.ts
  • src/lib/ai-edition/schema/index.ts
  • src/lib/ai-edition/timeline/zoom-scale.ts
  • technical-documentation/architecture/ai-agent.md
  • website/docs/ai-editing.md
  • website/i18n/de/docusaurus-plugin-content-docs/current/ai-editing.md
  • website/i18n/es/docusaurus-plugin-content-docs/current/ai-editing.md
  • website/i18n/fr/docusaurus-plugin-content-docs/current/ai-editing.md
  • website/i18n/ja/docusaurus-plugin-content-docs/current/ai-editing.md
  • website/i18n/pt-BR/docusaurus-plugin-content-docs/current/ai-editing.md
  • website/i18n/zh-CN/docusaurus-plugin-content-docs/current/ai-editing.md
  • website/i18n/zh-TW/docusaurus-plugin-content-docs/current/ai-editing.md

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 4 remain after this review.

📝 Walkthrough

The AI editing tools now support additional zoom settings, text, arrow, and blur annotations, and audio fades. The changes also update inspector controls, agent descriptions, tests, and documentation for these capabilities.

Changes: AI editing controls

Layer / File(s) Summary
Zoom settings and scale handling
electron/ai-edition/agent-tools.ts, electron/ai-edition/agent-tools.test.ts, electron/ai-edition/deep-agent/service.ts, src/lib/ai-edition/timeline/zoom-scale.ts, src/components/ai-edition/v4/FloatingInspector.tsx, technical-documentation/architecture/ai-agent.md
Zoom tools accept focus mode, cursor visibility, and custom scale. They map preset scales to depth values, reject conflicting or over-limit settings, and include the effective settings in tool results. The inspector uses the shared scale conversion.
Typed annotation creation and editing
electron/ai-edition/agent-tools.ts, electron/ai-edition/agent-tools.test.ts, electron/ai-edition/deep-agent/service.ts, src/lib/ai-edition/schema/index.ts, skills/openscreen-demo/SKILL.md, technical-documentation/architecture/ai-agent.md, website/docs/ai-editing.md, website/i18n/*/docusaurus-plugin-content-docs/current/ai-editing.md
Annotation tools support text, arrows, and blurs with type-specific geometry and data. They validate fields, convert annotation types, update shapes, and return type-specific readback. The agent guidance and documentation describe these annotation types.
Audio fades and shared limits
electron/ai-edition/agent-tools.ts, electron/ai-edition/agent-tools.test.ts, electron/ai-edition/deep-agent/service.ts, src/lib/ai-edition/document/audioTracks.ts, src/components/ai-edition/RightPanes.tsx, technical-documentation/architecture/ai-agent.md
Audio tools accept and update fade-in and fade-out durations, then report them in snapshots and results. The inspector and agent descriptions use the shared 5000 ms maximum.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to 9ffcb

This change adds zoom, annotation and audio-fade controls to the AI editing tools, with matching tests and documentation. No concrete merge-blocking risk was identified in the supplied context.

Pre-merge checks · Passed 5
Passed Passed checks (5 passed)
Check name Status Explanation
Description check Passed The description is complete and matches the template. It summarizes the feature, links the related issue, identifies the change type and release impact, states desktop impact, explains why screenshots…
Title check Passed The title clearly identifies the main change: region tools can set and read zoom, annotation, and audio settings.
Docstring Coverage Passed Docstring coverage is 94.12% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 9 files. (10 skipped: 1…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@EtienneLescot
EtienneLescot merged commit e896d77 into main Oct 13, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant