Skip to content

feat(nodes): add anthropic vision node - #673

Closed
ryan-t-christensen wants to merge 6 commits into
developfrom
feat/llm-vision-anthropic
Closed

ryan-t-christensen wants to merge 6 commits into
developfrom
feat/llm-vision-anthropic

Conversation

@ryan-t-christensen

@ryan-t-christensen ryan-t-christensen commented Apr 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Adds a new Anthropic Vision node (llm_vision_anthropic) that routes image frames through Claude Opus 4.6, Sonnet 4.6, and Haiku 4.5 with the same dual-lane architecture as the Gemini and Ollama vision nodes
  • Implements automatic JPEG compression for frames exceeding Anthropic's 5MB base64 string limit (not decoded size), with progressive quality reduction from 85→40 before skipping the frame entirely
  • Caps inference retries at 1 with a 30s hard timeout per attempt, matching the Ollama/Gemini pattern to prevent pipeline stalls

Type

feat

Testing

  • Tests added or updated
  • Tested locally
  • ./builder test passes

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included
  • Wiki updated (if applicable)
  • Breaking changes documented (if applicable)

Linked Issue

Fixes #

Summary by CodeRabbit

  • New Features

    • Added "Anthropic Vision" image-analysis node using Anthropic Claude vision models
    • Supports single-image and documents lanes, per-frame streaming, cached responses, and preserved metadata
    • Configurable model profiles, API key, system/user prompts, and UI profile selector
    • Returns text answers per image/document, enforces 5MB base64 limit with automatic JPEG compression and warnings, and improves timeout/retry/error handling
  • Documentation

    • Added user guide describing usage, model options, size limits, compression behavior, and setup instructions

@github-actions github-actions Bot added docs Documentation module:nodes Python pipeline nodes labels Apr 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

No description provided.

@coderabbitai

coderabbitai Bot commented Apr 14, 2026 •

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a new Anthropic Vision node: global lifecycle handler, Anthropic Chat integration, instance-level AVI image streaming and per-document processing, image size enforcement with progressive JPEG compression, threaded timeout/retry logic, service metadata, requirements, and README.

Changes

Cohort / File(s) Summary
Core runtime
nodes/src/nodes/llm_vision_anthropic/IGlobal.py, nodes/src/nodes/llm_vision_anthropic/IInstance.py, nodes/src/nodes/llm_vision_anthropic/anthropic_vision.py
Adds IGlobal lifecycle (loads local requirements, instantiates module Chat singleton); IInstance handles AVI BEGIN/WRITE/END streaming, frame buffering, per-frame cache logic, writeDocuments image→text processing and emit; Chat implements image base64 extraction, 5MB enforcement with progressive JPEG compression, threaded 30s SDK call, retry/backoff, and error mapping.
Package exports
nodes/src/nodes/llm_vision_anthropic/__init__.py
Re-exports IGlobal/IInstance, adds lazy getChat() and __all__.
Service config
nodes/src/nodes/llm_vision_anthropic/services.json
Adds image_vision_anthropic:// node definition, lanes (image/documents), model profiles, API key & prompt fields, and UI metadata.
Dependencies
nodes/src/nodes/llm_vision_anthropic/requirements.txt
Adds anthropic >=0.40.0 and pillow.
Docs
nodes/src/nodes/llm_vision_anthropic/README.md
New README documenting usage, supported models, 5MB base64 limit, progressive JPEG re-encoding behavior, and API key setup.

Sequence Diagrams

sequenceDiagram
    participant Client
    participant IInstance
    participant IGlobal
    participant Chat
    participant AnthropicSDK as Anthropic SDK

    Client->>IInstance: writeImage(BEGIN / WRITE... / END)
    IInstance->>IGlobal: request Chat (Question with base64 image)
    IGlobal->>Chat: chat(question)
    Chat->>Chat: enforce 5MB, compress if needed
    Chat->>AnthropicSDK: send vision request (daemon thread, 30s timeout)
    AnthropicSDK-->>Chat: response
    Chat-->>IGlobal: Answer
    IGlobal-->>IInstance: text result
    IInstance->>Client: emit text Doc (prevent default original image)
Loading
sequenceDiagram
    participant Client
    participant IInstance
    participant IGlobal
    participant depends
    participant Chat
    participant AnthropicSDK as Anthropic SDK

    Client->>IInstance: writeDocuments([Docs])
    IInstance->>IInstance: filter image Docs, build Question(base64)
    IInstance->>IGlobal: call _chat.chat(question)
    IGlobal->>depends: depends(requirements.txt)
    depends-->>IGlobal: dependencies loaded
    IGlobal->>Chat: forward chat call
    Chat->>AnthropicSDK: perform call (threaded, timeout, retry)
    AnthropicSDK-->>Chat: text response
    Chat-->>IInstance: Answer
    IInstance->>Client: emit text Docs (preserve metadata)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • jmaionchi
  • stepmikhaylov
  • Rod-Christensen

Poem

🐇 I nibbled bytes and stitched them tight,
I wrapped each pixel up in light,
A prompt I sent, a Claude that spoke,
I cached the answer, then danced a hop.
Vision decoded — carrot-coded delight!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 76.92% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The pull request title clearly and concisely describes the main change: adding a new Anthropic Vision node to the nodes module, which is the primary focus of all file additions in the changeset.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/llm-vision-anthropic

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nodes/src/nodes/llm_vision_anthropic/anthropic_vision.py`:
- Around line 60-63: Replace the current API key error messages with
non-revealing, generic messages and ensure no key value is included anywhere
when raising or logging: keep the existing checks (the api_key variable and the
startswith('sk-ant-') validation) but change the ValueError texts to generic
ones like "Missing Anthropic API key" and "Invalid Anthropic API key format"
and, if you need to include a representation, use a sanitizer function (e.g.,
sanitize_api_key) to mask the key before any logging or exceptions; also audit
any code that might catch these errors (or log api_key) to use the sanitized
value instead.

In `@nodes/src/nodes/llm_vision_anthropic/IInstance.py`:
- Line 41: The field _cached_answer on IInstance is annotated as str but
initialized to None; update its type to allow None (e.g., change to
Optional[str] or str | None) and add the corresponding import (from typing
import Optional) if using Optional[str]; ensure the annotation is updated on the
IInstance class so the default None value matches the declared type.
- Around line 91-92: The code is hardcoding "image/png" when building the data
URI for Question.addContext using doc.page_content; instead, read the actual
mime type from the Doc metadata (e.g., doc.metadata['mime_type'] or
doc.metadata['content_type']) and use that value in the data URI, falling back
to a safe generic like "image/*" (or detect format from the base64 payload) if
no metadata is present, and update the call in the Question/addContext
construction to use the dynamic mime type rather than the literal "image/png".
- Line 79: The type annotation for writeDocuments uses the Python 3.9+ syntax
list[Doc]; change it to typing.List[Doc] to maintain Python 3.8 compatibility:
update the signature of IInstance.writeDocuments to use List[Doc] and add or
ensure an import for List from typing at the top of the module (and keep the Doc
type reference unchanged).

In `@nodes/src/nodes/llm_vision_anthropic/README.md`:
- Around line 39-42: The fenced code block containing the lines "Anthropic
Vision: image is 5.4MB, exceeds 5MB limit — compressing to JPEG" and "Anthropic
Vision: compressed to 0.5MB at JPEG quality=85" needs a language specifier to
satisfy linting and enable proper highlighting; change the opening fence from
``` to ```text (i.e., replace the three backticks with ```text) so the block is
explicitly marked as plain text.

In `@nodes/src/nodes/llm_vision_anthropic/requirements.txt`:
- Line 2: Update the Pillow requirement in
nodes/src/nodes/llm_vision_anthropic/requirements.txt to exclude vulnerable
releases by replacing the current declaration "Pillow>=10.0.0" with
"Pillow>=10.3.0"; after changing the requirement, regenerate any dependency
lock/constraints (pip-compile, poetry lock, or pipenv lock) and run the
project's test suite or a quick import check to ensure the newer Pillow is
compatible with functions in this node that use PIL imaging.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6a278f55-6d39-4f13-bc9d-615b50a8a8b4

📥 Commits

Reviewing files that changed from the base of the PR and between 0ac4937 and d3fa0f8.

📒 Files selected for processing (7)
  • nodes/src/nodes/llm_vision_anthropic/IGlobal.py
  • nodes/src/nodes/llm_vision_anthropic/IInstance.py
  • nodes/src/nodes/llm_vision_anthropic/README.md
  • nodes/src/nodes/llm_vision_anthropic/__init__.py
  • nodes/src/nodes/llm_vision_anthropic/anthropic_vision.py
  • nodes/src/nodes/llm_vision_anthropic/requirements.txt
  • nodes/src/nodes/llm_vision_anthropic/services.json

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nodes/src/nodes/llm_vision_anthropic/IInstance.py`:
- Around line 43-77: The writeImage method lacks handling for unexpected
AVI_ACTION values; update writeImage (in IInstance.writeImage) to add a final
else/guard that explicitly handles unknown actions by raising a RuntimeError (or
returning a clear error) including the invalid action value and context (e.g.,
mimeType) so callers get a deterministic failure instead of falling through;
reference AVI_ACTION and the writeImage function when adding this defensive
branch.
- Around line 36-41: Replace the Python 3.10+ union syntax with typing.Optional
for compatibility: change the type annotations on image_data and _cached_answer
in IInstance (symbols: image_data and _cached_answer) from "bytearray | None"
and "str | None" to "Optional[bytearray]" and "Optional[str]" and add "from
typing import Optional" at the top if not present so the file remains compatible
with Python 3.8+.
- Line 99: The warning message uses doc.metadata.chunkId without null-checking
metadata; update the code in IInstance.py where the warning is emitted (the line
calling warning(...)) to safely handle missing metadata by deriving chunk_id =
doc.metadata.chunkId if doc.metadata is not None (or using getattr/doc.metadata
and a default like "unknown") and interpolate chunk_id into the warning string
so accessing chunkId cannot raise AttributeError.

In `@nodes/src/nodes/llm_vision_anthropic/requirements.txt`:
- Line 2: Update the unconstrained "pillow" dependency entry so it requires a
safe minimum version; replace the bare "pillow" requirement with
"Pillow>=10.3.0" to exclude known vulnerable releases (update the requirements
entry that currently just says "pillow"). Ensure the package name uses the
canonical capitalization "Pillow>=10.3.0" to make the constraint explicit in the
manifest.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 0ff67c77-c750-492c-9482-a1abf5b3cd73

📥 Commits

Reviewing files that changed from the base of the PR and between d3fa0f8 and ee254cc.

📒 Files selected for processing (3)
  • nodes/src/nodes/llm_vision_anthropic/IInstance.py
  • nodes/src/nodes/llm_vision_anthropic/README.md
  • nodes/src/nodes/llm_vision_anthropic/requirements.txt

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nodes/src/nodes/llm_vision_anthropic/IInstance.py`:
- Around line 63-75: The image-only path in writeImage calls
self.IGlobal._chat.chat(question) without error handling; wrap the chat call and
subsequent answer.getText() in a try/except, mirror the graceful degradation
used in writeDocuments: on exception log the error (use the same logger used
elsewhere, e.g., self.IGlobal._log or the project's process logger), and call
self.instance.writeText with a safe fallback (empty string or a short error
message) instead of letting the exception escape; keep the Question creation and
image_data_url logic, but ensure self.instance.writeText is always called even
if the chat call fails.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6d2481a7-3ae3-415c-80c2-7b7fd5182401

📥 Commits

Reviewing files that changed from the base of the PR and between ee254cc and 9d97591.

📒 Files selected for processing (1)
  • nodes/src/nodes/llm_vision_anthropic/IInstance.py

joshuadarron
joshuadarron previously approved these changes Apr 15, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nodes/src/nodes/llm_vision_anthropic/IInstance.py`:
- Around line 134-135: The writeDocuments path calls self.preventDefault() but
does not return its result, unlike writeImage which returns
self.preventDefault(); update the writeDocuments implementation to return
self.preventDefault() so the prevention signal is propagated consistently
(locate the writeDocuments method in IInstance.py and replace the bare
self.preventDefault() call with a return of self.preventDefault()).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 55d4caeb-b4d5-4b81-ba26-36d66c5d90e9

📥 Commits

Reviewing files that changed from the base of the PR and between c446047 and cc7bfed.

📒 Files selected for processing (1)
  • nodes/src/nodes/llm_vision_anthropic/IInstance.py

@ryan-t-christensen
ryan-t-christensen marked this pull request as draft April 16, 2026 02:10
@stepmikhaylov
stepmikhaylov deleted the feat/llm-vision-anthropic branch September 22, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation module:nodes Python pipeline nodes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants