[JamesMowery]

Important

Contributions target staging. Only SpicyMarinara may promote this repository's staging branch or a same-repository hotfix/* branch to main.
Outside and first-time contributors also require an approving review from SpicyMarinara.

Note

#7860 and #7863 together, as asked. Both were closed for the 2.5.1 release with a request to bring them back after it, with the review fixes. This is the #7863 branch (which already contained #7860) merged with the current staging (4d2932c05), plus the fixes listed under Review fixes below.

Linked issue

Closes #7992
Part of #7545 (item 6 of 8). Replaces #7860 (#7859) and #7863 (#7862). @SpicyMarinara

Why this change

• Videos only happened when you asked for one. In Conversation a character could send a selfie on its own but never a video message.
• In Roleplay, Illustrator drew pictures on its own but could never make a short clip, even for a moment where motion is the point (a fight, a chase, something breaking), and there was no way to have clips instead of pictures.
• Clips made on their own need limits: each is a video request that can take minutes and cost money or GPU time. With Clips only, Illustrator was paced only by the image interval, and in a group chat only the first character's [video] was made and the rest were dropped silently.

What changed

Nothing that worked before stops working: Generated Videos starts off, Illustrator makes starts on Pictures, and the interval Illustrator already had is now labeled Image Run Interval and behaves as before.

Conversation

• Generated Videos (Chat Settings, Agents, Illustrator Settings, Video messages; off by default, saved in settings profiles): characters can send a video message on their own with a hidden [video] command when it fits. The same switch appears as Videos in the Commands list of the new-chat setup and Chat Settings.
• A bare [video] is written from the recent conversation, as /video is, so a clip you asked for shows what you asked for.
• A [video: …] description of any length is read, brackets of its own included ([Shot 1] …); one over 2,000 characters is shortened by the Prompt Model, or cut at 2,000 if that fails. A tag that never closes makes no clip and is hidden to the end of its line.
• A clip that names you describes you (your persona, or a character you play as who is not one of the chat's characters), and a clip that names your persona is listed in your persona's gallery.

Roleplay (Chat Settings, Agents, Illustrator)

• Illustrator makes: Pictures (the default), Pictures and clips (Illustrator may turn the picture it just made into a short clip when motion is the point; it is skipped when Illustrator doesn't say how the scene moves), or Clips only (a clip instead of every picture; the picture is the clip's starting frame and goes to the Gallery, and is shown instead if the clip fails). Custom agents that create images follow the same setting.
• First frame is shown in Roleplay too: a new Illustrator picture, the avatar or sprite, the picture with the avatar as fallback, or no picture. With Clips only and the avatar or no picture, no picture is drawn at all. With no First frame chosen, a clip never draws a picture just for itself.
• Send Avatar References becomes a dropdown with a Videos choice: a Prompt only clip then starts from the avatar, and First frame reads Prompt only (avatar from Send Avatar References). Use agent default resets both halves.
• A clip with no picture of its own is written by a new Roleplay Clip Director prompt (Settings, Generations, Video Generation Prompt Overrides), describing everyone it names, your persona included.
• Re-running Illustrator on a reply (its Illustrate button) follows Illustrator makes and First frame too. Before, a re-run only ever drew a picture.
• Without a working video connection, Illustrator makes pictures, even with Clips only, and is not offered clips.

Limits (only for clips the AI makes on its own, never ones you ask for)

• Video Run Interval: how many user and assistant messages must pass after an automatic clip before the next.
  • Conversation: in Chat Settings, Video messages, shown once Generated Videos is on.
  • Roleplay: on the Illustrator card, shown once Illustrator makes includes clips. With Clips only it paces Illustrator itself and the Image Run Interval is hidden.
  • Default on the Illustrator card's Open Setup screen (5 when unset); 0 stops automatic clips. Saved in settings profiles.
  • Never limits /video, the Gallery's Video and Animate, or a message's Illustrate button.
  • A clip is only offered to the AI when one may be made now, so a character isn't told to send a clip that would then be refused.
  • Counted over the whole chat, even when a short context window hides the last clip, and from when a clip starts rather than when it finishes minutes later. Replies that arrive while a clip renders can't start another, two replies finishing together can't both start one, and a clip that fails gives the interval back.
  • Swiping a reply that got a clip counts as a new reply, so the new swipe can get its own clip.
• Videos per reply (Settings, Generations, Video Generation; 1 to 10, default 3; findable in Settings search): how many clips one reply gets, one per Conversation character. It exists for group chats and for asking for several videos at once; lowering it saves video requests. Clips past it are skipped and logged, and a failed clip gives its place back.
• The storyteller's [illustrate] clips (Clips only, or Pictures and clips) go through the same interval check and count toward Videos per reply; one held back shows its picture instead.

Every clip

• Follows the video connection's Video Prompting Instructions: format, fields, line breaks, and a JSON shape they ask for. When the instructions speak to appearance, it is built into the prompt.
• A saved clip always counts as made: if adding it to the reply fails, it stays in the Gallery with a "saved to the Gallery's Videos tab" toast, and no error or stand-in picture.
• A reply whose only remaining work is a clip lets the next message start (video_started / video_skipped), as a pending picture does.
• Pressing Stop reaches the clips: the queue wait, the Clip Director call, the first frame and the video request. A stopped clip shows no error.

Code: the clip pieces shared by a fresh reply and a re-run (who a clip shows, its prompt, the clip offer to Illustrator) live in services/video/roleplay-illustrator-clips.ts; the <optional_video> offer text is shared by the Illustrator agent and the Illustrate planner; the interval claim is claimAutomaticVideoRun in services/generation/agent-cadence.ts.

Docs: docs/agents/built-in-agents.md, docs/conversation/selfies.md, docs/media/illustrator-agent.md and docs/media/scene-video.md.

Review fixes (from #7860 and #7863)

• Land with feat(video): Video Run Interval and Videos per reply for automatic clips #7863: one PR now.
• Avatar first frame with short names (bc855b069): the first name is resolved on its own with reference images, so "Lena" finds "Dr. Lena Kowalski"; roleplay-clip-first-frame covers a multi-word card name.
• Stop reaching the clips (bc855b069): the agent signal goes to createRoleplayClipTools, animateGalleryImage, generateRoleplayClip (new signal) and startConversationVideoCommand / generateConversationVideo; the re-run passes its own signal. It is checked up to the provider request and passed to it.
• [illustrate] clips skipping the interval (bc855b069, 97a714760): both paths go through claimRoleplayClip (the interval claim with the same allowed check as automatic clips, plus the Videos per reply cap); a refused one draws its picture. roleplay-clip-first-frame is updated.
• The UI test the new input trips (bc855b069): the Image Run Interval input has its own aria-label, and e2e/illustrator-manual-cadence.e2e.ts uses getByRole("spinbutton", { name: "Image Run Interval" }). Not run locally (see Validation).
• agent-cadence.ts catch: returns null when allowed was passed, so a refused clip can't slip through on a write error.
• Docs pointer: illustrator-agent.md names the Illustrator card's Open Setup screen.

Dottore review fixes (2026-10-22)

• A clip animated from a picture is filed under the picture's cast (and the persona), not the whole chat, on a fresh reply and an Illustrate re-run.
• Conversation [video] is offered only when the chat has a usable video connection (hasChatVideoConnection), as [selfie] needs an image connection.
• A Conversation clip that fails gives back its Videos per reply slot and its character's one clip.
• An Illustrate re-run reads the chat settings it already parsed safely, so malformed metadata can't end the run.
• Only a JSON shape the video connection's instructions ask for (they name JSON or show an object) becomes a clip's prompt; a stray object such as an error payload doesn't.
• built-in-agents.md names Illustrator makes for Roleplay clips; one shared isConversationCommandToggleOn holds the Videos opt-in rule.

Validation

• pnpm check passes locally
• Container (Docker / Podman) built and ran without issue
• Ran the app, clicked through the changes manually
• Checked edge cases (light + dark mode, mobile viewport, empty states, error paths)
• Above manual verification completed (describe below)
• Read and followed CONTRIBUTING.md

Automated runs on 2026-10-21, Node 24.21.0:

• On cb908755a (staging 4d2932c05 merged in, no conflicts):
  • pnpm check: passes. Its one lint warning, in HomeProfessorMariChat.tsx, is not from this PR.
  • pnpm localization:check: passes.
  • Full pnpm regression: 960/960 pass.
• On d6be94a30 (local CodeRabbit fixes): pnpm check passes; agent-generated-videos, illustrator-video-run-interval, roleplay-clip-first-frame, convo-video-command, capability-gm-verbs and prompt pass.
• Regressions added by this PR, through the real routes with fake text, image and ComfyUI services:
  • agent-generated-videos: Conversation [video] on and off, the bare [video], bracketed and unclosed tags, Pictures and clips and Clips only, a failed clip showing its picture, a saved-but-unattached clip counting as made, persona tagging, a character played as the user, appearance built in when the instructions ask for it, Videos per reply and long descriptions.
  • roleplay-clip-first-frame: the First frame choices, a short name finding its card's avatar, Send Avatar References for videos, the Clip Director and instruction-shaped answers, no working video connection, re-running Illustrator, and [illustrate] clips inside and after the interval.
  • illustrator-video-run-interval: both intervals, a short context window, agent default and chat override, 0, swipes, and the interval claim (racing replies, clips sharing one message, a failed write).
  • The new checks fail without their fixes.
• Local CodeRabbit (CLI) on the whole branch against staging 4d2932c05: 11 findings.
  • Fixed in d6be94a30: a stopped Roleplay clip reported as a 502 "failed to plan" and a video_error (two findings); a re-run whose clip failed ending with "Agent retry completed"; debug mode not logging the text sent to be shortened; the Video Run Interval paragraph in illustrator-agent.md split into a list.
  • Declined: 16px text on phones for the new number input and selects, and making the globals.css touch-screen rule 16px (three findings). That rule already sets inputs and selects to 1rem under @media (pointer: coarse); it falls short only with a smaller app font size, which affects every input in the app, not this PR. Declined: tryParseJsonRecord throwing in resolveRoleplayVideoDirection, because repairJsonText only returns text JSON.parse already accepted. Declined: splitting two docs paragraphs that are already short sentences.
  • A second review of d6be94a30: 0 findings.
• Not run: e2e/illustrator-manual-cadence.e2e.ts and the rest of pnpm regression:ui / pnpm smoke:ui. Playwright's Chromium download from cdn.playwright.dev times out on this machine, so the spec the review asked to run is left to CI or a manual run.

Manual verification notes

• Manually run pnpm exec playwright test e2e/illustrator-manual-cadence.e2e.ts and confirm it passes.
• Manually verify in a Conversation chat: turn on Generated Videos (or Videos in Commands), ask a character for a video, and check the clip lands on their reply; with it off, no clip arrives and /video still works.
• Manually verify a clip that names your persona lists in your persona's gallery, and with Attach Card Appearance on Videos, its prompt carries your appearance.
• Manually verify Conversation with Video Run Interval 3: a clip, then no clip on the next reply, then clips again after three messages; with 0, no automatic clips while /video and the Gallery's Video button still work.
• Manually verify a group Conversation with Videos per reply on 3: asking everyone for a video gives up to three clips, one per character; on 1, only one.
• Manually verify Roleplay Pictures and clips: an action moment gives a picture, then a clip on the same reply; pictures keep the Image Run Interval while clips keep the Video Run Interval.
• Manually verify Clips only with First frame New Illustrator picture (clip in the reply, picture in the Gallery only), then Avatar with a character addressed by a short name, and Prompt only (no picture drawn).
• Manually verify Clips only with Video Run Interval 0: a storyteller [illustrate] gives its picture, not a clip, while a message's Illustrate button still makes a clip.
• Manually verify pressing Stop while a Roleplay clip is waiting or being written cancels it with no error toast.
• Manually verify Clips only with no working video connection gives pictures and no error.
• Manually verify swiping a reply that got a clip gives the new swipe its own clip and keeps the old one.
• Manually verify a connection whose Video Prompting Instructions ask for a multi-field or JSON format keeps that format in the prompt sent.
• Manually verify the new settings at phone width and in light and dark mode.

Docs and release impact

• No docs changes needed
• Updated docs (README / CONTRIBUTING / android/README) as needed
• Translated docs on the docs-i18n branch updated to match, or a [docs-i18n] follow-up issue opened (see CONTRIBUTING.md § Translated documentation)
• Version/release files updated (only if this PR includes a version bump)

English docs updated: docs/agents/built-in-agents.md, docs/conversation/selfies.md, docs/media/illustrator-agent.md, docs/media/scene-video.md. Translations: #7994. The Marinara-Agents README follow-up: Pasta-Devs/Marinara-Agents#1492.

UI evidence (if applicable)

From #7860 and #7863 (taken 2026-10-19; the settings are unchanged since).

Roleplay mode illustrator agent additions

"Pictures and Clips" illustrator setting demo

(Note: I did not have an I2V workflow activated here in ComfyUI, thus the visual difference between the image and video.)

"Clips Only" illustrator settings demo

Videos command toggle

Convo mode: requesting a clip with Videos toggle enabled

Convo mode: requesting a clip with Videos toggle disabled (reverts to image)

Videos per reply setting in Settings

Video Run Interval and renamed Image Run Interval

Videos per reply setting at default (3)

Videos per reply setting set to 2

🤖 Generated with Claude Code

[JamesMowery]

@SpicyMarinara Dottore's run here didn't finish on the reviewer's side (the model timed out after 600 s), so the PR hasn't been reviewed yet. Could you comment /dottore to run it again? My account can't trigger it.

[SpicyMarinara]

/dottore

[github-actions]

Dottore Review — isolated defects from the specimen.

[github-actions]

⚠️ MEDIUM: Send Avatar References keeps a second control that cannot see the video half

Location: packages/client/src/components/chat/ChatSettingsDrawer.tsx:9276

One setting now wears two faces in the same drawer: the four-way dropdown that writes illustratorUseAvatarReferences and illustratorVideoUseAvatarReferences together, and the legacy toggle a few thousand lines below whose mutation carries only the image key. Switch that toggle off and the video flag stays exactly where the dropdown left it, so a Prompt only clip keeps starting from the avatar — roleplayVideoFirstFrame reads chatMeta.illustratorVideoUseAvatarReferences === true and returns avatar_or_none — while the control that claims to govern references reads off. The reset has already been taught to clear both halves, which makes the asymmetry worse: a control displaying one half silently discards the other, and a chat set to Videos only through the dropdown shows the ca

[truncated: inline finding was 1433 chars, limit is 1085 chars]

🎭 Push a repair and Dottore re-examines this thread; or tick the 🩺 box in his walkthrough to let him repair it.

[JamesMowery]

Not changed. The two controls never appear in the same chat. The four-way Send Avatar References dropdown that writes both keys is in the Roleplay Illustrator Settings (rendered only when isRoleplayMode && illustratorActive). The on/off toggle in this agent card is in the Game Mode branch (isGame ? … gameAgentPool). Game Mode's scene videos never read illustratorVideoUseAvatarReferences; only the Roleplay clip paths (roleplayVideoFirstFrame and the Illustrator clip cast) do. So a Game chat's image-only toggle can't leave a video flag active anywhere, and a Roleplay chat never shows that toggle.

[github-actions]

ℹ️ LOW: [video] is advertised to characters with no video connection behind it

Location: packages/server/src/services/generation/conversation-command-runtime.ts:276

Directly above, the selfie offer waits on hasImageGen — a connection the chat actually picked — before it promises a photo. The new video offer waits on nothing of the kind: videoCommandEnabled is isConversationCommandAvailable, a room allowance, the toggle, and the Video Run Interval, none of which knows whether a video-generation connection exists. A chat with Generated Videos turned on and no usable connection therefore instructs the character to send clips, and each one runs generateConversationVideo into resolveChatVideoSetup, which throws ChatVideoError(400, "Choose a video connection in Chat Settings before making a video.") and returns to the reply as video_error. The illustration side already knows better: resolveIllustratorVideoOffer returns null without a work

[truncated: inline finding was 1320 chars, limit is 1085 chars]

🎭 Push a repair and Dottore re-examines this thread; or tick the 🩺 box in his walkthrough to let him repair it.

[github-actions]

🎭 Dottore cannot discharge this specimen yet: its examination of this head did not end Viable; 7 Dottore thread(s) are unresolved; these checks have not passed: Owner approval for outside contributors, Pull Request template check.

[JamesMowery]

Thanks for the re-run, @SpicyMarinara. Dottore's review is addressed in 5f438cb85 to df781a9d4, with staging merged in first (a9468b06f; conflicts only around staging's per-chat card fields, carried over as getPersona(…, chatId) in the re-run's clip cast).

• A clip animated from a picture took the whole chat's cast: fixed. Both picture branches (a fresh reply and an Illustrate re-run) now pass the picture's own cast and the persona. roleplay-clip-first-frame animates a picture of Aster in a chat with Aster and Bram and expects the clip filed under Aster only.
• [video] offered with no video connection: fixed. The Conversation offer now also needs a usable video connection, found the way clip-making finds it (hasChatVideoConnection, via getWithKey). agent-generated-videos checks it isn't offered when the chat's video connection isn't a video connection, and is offered with a real one.
• A failed Conversation clip kept its quota: fixed. startConversationVideoCommand resolves to whether the clip was made, and a failed one gives back its Videos per reply slot and its character's one clip, as Roleplay clips do. No dedicated test: a failure landing between two [video] commands of one reply is timing-dependent, and the group-clip regressions pass.
• The re-run re-parsed the metadata column raw: fixed; it uses the chatMeta parsed safely at the top of the handler.
• A stray JSON object became the video prompt: fixed. Only a JSON shape the instructions ask for (they name JSON or show an object) is the prompt. prompt covers plain instructions with {"error":"rate limited"}.
• Built-in agents doc credited the wrong switch: fixed; it names Illustrator makes (labels checked against ui.chat.chatsettingsdrawer.illustratorMakes and the two mode strings).
• Send Avatar References' second control: not changed; reasoning in the thread.
• Nitpick, opt-in rule written twice: fixed with one shared isConversationCommandToggleOn in @marinara-engine/shared.
• Nitpick, help text above its select: left as is; it is the order the shared MediaScopeSelect uses (label, help, select).
• Localization warnings: pnpm localization:check passes, and "Image Run Interval" / "Video Run Interval" are exact catalog values (ui.agents.agenteditor.imageRunInterval / .videoRunInterval).

On f404cfa09: pnpm check and pnpm localization:check pass; full pnpm regression: 1002/1002. Local CodeRabbit on the fixes: two applied in df781a9d4 (getWithKey, shorter doc sentences). Declined: a package-root import (both files already import from @marinara-engine/shared's root), and more mode/group cases for the [video] offer (the gate only applies to Conversation, and the Conversation cases are covered).

[SpicyMarinara]

Reviewed df781a9d4e0727b6d4457139357471051d4448b7.

Changes requested: make malformed video-tag parsing linear before merging.

At packages/server/src/services/conversation/character-commands.ts:470–486, every unclosed [video: scans to the end of the response, but failure advances only to that line's end. The next malformed line scans almost the same suffix again. parseCharacterCommands performs this scan for both extraction and stripping, before the later command/video-connection checks.

I independently reproduced this through the actual exported parser with '[video:\n'.repeat(n): 16 KB took 83 ms, 32 KB 310 ms, 64 KB 1.30 s and 128 KB 7.34 s on this machine. The pre-feature parser took 3.37 ms for the same 128 KB input; a larger balanced-tag control took 21 ms. Timing varies by host, but the repeated suffix scan directly establishes quadratic server event-loop work on model-controlled text.

Please preserve balanced multiline descriptions and the existing malformed-line fallback while ensuring previously examined suffixes are not rescanned. Add a scaling regression with malformed multiline tags and balanced/multiline controls.

The feature is coherent host integration and should remain open for repair. I also verified that the old second-avatar-reference-control finding is a mode-separation false positive (Game-only versus Roleplay-only), rather than treating all historical findings as current defects.

[github-actions]

Dottore Review — isolated defects from the specimen.

[JamesMowery]

Fixed in caa9ff1. readVideoTags now pairs every bracket once with a stack before reading tags, so an unclosed [video: never rescans the rest of the reply. Balanced multiline descriptions and the hide-to-end-of-line fallback for a tag that never closes are unchanged.

agent-generated-videos adds:

• a scaling check: 8× the malformed text must take well under 64× as long (best of 3 runs each)
• an unclosed tag followed by a closing tag on the next line, which still makes its clip
• malformed lines before a balanced multiline tag, which stays whole

The old parser didn't finish the new scaling check within 280 s; the fixed one passes the whole regression in about 20 s.

[SpicyMarinara]

Thank you!