Skip to content

feat(prime): analysis of image-model routing (#2453) vs composer model selection and quota tracking #725

Description

@rynfar

Background & Upstream Change

Upstream Prime Agent PR #2453 introduced sub-agent image-model routing:

  • When sessions encounter image assets or multimodal tasks, sub-agents can be spawned or routed to an alternate, vision-capable model (or a cheaper vision model) instead of strictly inheriting the parent conversation's model.

Analysis for Pylon

1. Interaction with Composer Model Selector

  • In Pylon, the user explicitly selects a primary model via the composer selector, or relies on workspace defaults.
  • If image-model routing is strictly agent-internal (transparently dispatched by the agent runtime when inspecting images), it would run behind the scenes without overriding the user's primary composer model.
  • However, if the user explicitly chose an expensive or specific model, silent fallback to an unconfigured or unapproved provider/model for vision could lead to unexpected behavioral differences or privacy/compliance leaks (e.g. sending images to an external third-party vision provider).

2. Interaction with Quota Tracking & Accounting

  • Pylon tracks provider token usage and quotas per instance/workspace.
  • If sub-agents route images to a secondary model:
    • Token accounting and spend calculations must be correctly attributed to the secondary model's rates in the journal/ledger.
    • Quota exhaustion on the image model must not wedge the primary conversation or corrupt turn settlement.

3. UI and Configuration Exposure Recommendation

  • Phase 1 (Config-only / Agent-internal):
    • Expose an optional setting in provider settings (e.g. imageProcessingModel or visionModel), similar to upstream's auxiliaryModel or subagentDefaultModel.
    • When unset, default to the active session model (preserving existing behavior).
  • Phase 2 (UI indicators):
    • In the session inspector / subagent view, indicate when an image-processing subagent ran with an alternate model.
    • Do not complicate the primary composer model picker with auxiliary sub-pickers unless the user enters a specialized multimodal inspection mode.

Action Items

  • Evaluate adding an imageModel configuration field to Pylon's Prime provider settings.
  • Ensure token usage accounting tracks multi-model costs accurately in session records.
  • Define fallback behavior when the configured vision model fails authentication or hits quota.

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions