Skip to content

feat: stream llm output to the UI - #781

Closed
AditM99 wants to merge 3 commits into
rocketride-org:developfrom
AditM99:feat/RR-730-stream-llm-output
Closed

AditM99 wants to merge 3 commits into
rocketride-org:developfrom
AditM99:feat/RR-730-stream-llm-output

Conversation

@AditM99

@AditM99 AditM99 commented May 7, 2026 •

Copy link
Copy Markdown

Summary

Streams the LLM output to the user so user does not need to wait to see the output until the entire output is generated

Type

feature

Testing

./builder test passes

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included
  • Wiki updated (if applicable)
  • Breaking changes documented (if applicable)

Linked Issue

Fixes #730

Summary by CodeRabbit

  • New Features

    • Real-time streaming enabled across many LLM providers and core chat, delivering progressive responses.
    • New streaming API returns ordered, sequenced intermediate chunks (SSE) while producing a final consolidated answer.
    • Streaming automatically falls back to non‑streaming for JSON/structured-output requests.
  • Bug Fixes / Resilience

    • Improved retry handling and explicit terminal/error chunk emission to ensure robust streaming recovery.

@github-actions github-actions Bot added module:nodes Python pipeline nodes module:ai AI/ML modules labels May 7, 2026
@coderabbitai

coderabbitai Bot commented May 7, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack
No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 68d429ee-f16f-49fd-9e94-a60e029b0f91

📥 Commits

Reviewing files that changed from the base of the PR and between 44f0181 and fc13973.

📒 Files selected for processing (1)
  • packages/ai/src/ai/common/chat.py

📝 Walkthrough

Walkthrough

Adds streaming support: ChatBase exposes a chat_stream API with retry semantics; IInstance routes to a streaming path that emits SSE lm.chunk events; several providers declare SUPPORTS_STREAMING and multiple providers implement provider-specific _chat_stream methods.

Changes

LLM Streaming Support

Layer / File(s) Summary
ChatBase Streaming Infrastructure
packages/ai/src/ai/common/chat.py
Adds SUPPORTS_STREAMING = False on ChatBase, default _chat_stream(prompt, on_chunk), _chat_stream_with_retries(...) that stops retrying after first chunk, and public chat_stream(question, on_chunk) returning an accumulated Answer.
Instance Orchestration & SSE Emission
nodes/src/nodes/llm_base/IInstance.py
_question now conditionally routes to _question_stream when provider SUPPORTS_STREAMING and expectJson is false; _question_stream calls chat_stream, serializes SSE llm.chunk emissions under a per-call lock, tags chunks with nodeId and monotonic seq, emits terminal done/error chunk, and returns the final Answer.
Provider Streaming Implementations
nodes/src/nodes/llm_gemini/gemini.py, nodes/src/nodes/llm_ibm_watson/ibm_watson.py, nodes/src/nodes/llm_mistral/mistral.py
Gemini, IBM Watson, and Mistral add _chat_stream(prompt, on_chunk) implementations that iterate provider streaming APIs, extract text deltas, invoke callback per chunk, and return concatenated output. Typing imports updated to include Callable where required.
Provider Streaming Capability Declarations
nodes/src/nodes/llm_anthropic/anthropic.py, nodes/src/nodes/llm_bedrock/bedrock.py, nodes/src/nodes/llm_deepseek/deepseek.py, nodes/src/nodes/llm_gmi_cloud/gmi_cloud.py, nodes/src/nodes/llm_ollama/ollama.py, nodes/src/nodes/llm_openai/openai_client.py, nodes/src/nodes/llm_openai_api/openai_client.py, nodes/src/nodes/llm_perplexity/perplexity.py, nodes/src/nodes/llm_qwen/qwen_client.py, nodes/src/nodes/llm_vertex/vertex.py, nodes/src/nodes/llm_xai/xai.py
Multiple provider Chat classes now declare SUPPORTS_STREAMING = True to advertise streaming capability to the instance orchestration layer.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant IInstance
  participant ChatBase
  participant Provider
  Client->>IInstance: ask(question)
  IInstance->>IInstance: check SUPPORTS_STREAMING & expectJson
  alt streaming path
    IInstance->>ChatBase: chat_stream(question,on_chunk)
    ChatBase->>Provider: _chat_stream_with_retries(prompt,on_chunk)
    Provider->>ChatBase: on_chunk(text) (multiple)
    ChatBase->>IInstance: on_chunk forwarded
    IInstance->>Client: sendSSE(llm.chunk,nodeId,seq)
    Provider-->>ChatBase: final text
    ChatBase-->>IInstance: Answer
    IInstance-->>Client: final Answer
  else non-streaming path
    IInstance->>Provider: chat(question)
    Provider-->>IInstance: Answer
    IInstance-->>Client: Answer
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • jmaionchi
  • stepmikhaylov
  • Rod-Christensen

Poem

🐰 Streams of tokens hop in flight,
Chunks arrive, a glowing sight.
Callbacks hum and SSE sings,
Words unfurl on digital wings.
Rabbit cheers: stream on tonight!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'feat: stream llm output to the UI' directly aligns with the main objective to implement incremental streaming of LLM outputs to the user interface.
Linked Issues check ✅ Passed The PR implements core streaming infrastructure across all 16+ LLM providers and base chat framework, enabling streaming execution paths and SSE-based chunk emission as required by issue #730.
Out of Scope Changes check ✅ Passed All changes are within scope: LLM provider streaming flags, base chat streaming methods, and router logic in IInstance to conditionally use streaming. No tool-call streaming, history persistence, or unrelated changes detected.
Docstring Coverage ✅ Passed Docstring coverage is 87.10% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented May 7, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
nodes/src/nodes/llm_xai/xai.py (1)

60-62: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Remove duplicate super().__init__ call in constructor.

ChatBase is initialized twice (Line 48 and Line 61). This can re-run base setup and overwrite state unexpectedly.

Proposed fix
         # API key validation logic
         if not apikey or not apikey.startswith('xai-'):
             raise ValueError('Invalid XAI API key format, please check your API key.')
-
-        # Init the chat base
-        super().__init__(provider, connConfig, bag)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/llm_xai/xai.py` around lines 60 - 62, The constructor calls
the base class initializer twice which re-runs ChatBase setup and can overwrite
state; remove the duplicate super().__init__(provider, connConfig, bag) so
ChatBase is only initialized once (keep the original call where the base is
first initialized and delete the later redundant call in the XAI constructor
that re-invokes super().__init__).
nodes/src/nodes/llm_ollama/ollama.py (1)

57-60: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Guard serverbase before calling .rstrip().

config.get('serverbase', None) can return None, and None.rstrip(...) will raise AttributeError.

Proposed fix
-        serverbase = config.get('serverbase', None)
+        serverbase = config.get('serverbase')
+        if not serverbase:
+            raise ValueError('Ollama serverbase is required.')

         if not serverbase.rstrip('/').endswith('/v1'):
             serverbase = serverbase.rstrip('/') + '/v1'
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/llm_ollama/ollama.py` around lines 57 - 60, The code calls
serverbase.rstrip(...) without guarding against serverbase being None (from
config.get('serverbase', None)); update the logic in ollama.py to check that
serverbase is truthy before calling rstrip (e.g., use "if serverbase and not
serverbase.rstrip('/').endswith('/v1'):"), leaving serverbase unchanged when
None or empty, so you avoid AttributeError while preserving the existing
normalization behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@nodes/src/nodes/llm_base/IInstance.py`:
- Around line 69-77: The except block in IInstance.chat_stream is redundantly
calling map_exception on an exception that _chat_stream_with_retries already
mapped, causing a self-referential exception chain; update the except handler to
stop remapping (remove the second call to self.IGlobal._chat.map_exception) and
simply emit(done=True, error=str(e)) and re-raise the original exception (raise)
so that you don't set e.__cause__ = e; keep emit usage and references to
self.IGlobal._chat, chat_stream, _chat_stream_with_retries and map_exception
intact elsewhere.
- Line 70: The call to chat_stream wraps emit in an unnecessary lambda; replace
the wrapper by passing the emit function directly to
self.IGlobal._chat.chat_stream (i.e., change the second argument from "lambda
delta: emit(delta)" to "emit") since emit(text, done=False, error=None) already
accepts a single positional text argument and matches the expected callback
signature.

In `@nodes/src/nodes/llm_gemini/gemini.py`:
- Line 24: Replace deprecated typing aliases: change the import line to only
import Any from typing and import Callable from collections.abc (e.g., "from
typing import Any" and "from collections.abc import Callable"), then update all
type annotations that use Dict to use the built-in dict (e.g., dict[str, Any] or
dict[Any, Any] as appropriate); ensure no other uses of
typing.Callable/typing.Dict remain in gemini.py.

In `@nodes/src/nodes/llm_ibm_watson/ibm_watson.py`:
- Line 25: Replace deprecated typing aliases in the module import: stop
importing Callable and Dict from typing; import Callable from collections.abc
and use the built-in dict type for annotations instead of typing.Dict. Update
the import line in ibm_watson.py to keep Any from typing (if needed) and ensure
all function/type annotations that used Dict[...] are changed to dict[...], and
annotations using Callable use collections.abc.Callable.

In `@nodes/src/nodes/llm_mistral/mistral.py`:
- Line 40: Replace deprecated typing aliases on the import line by importing
Callable from collections.abc and using built-in dict and tuple types: update
the import statement that currently brings in Any, Callable, Dict, Tuple from
typing, and change Callable to be imported from collections.abc while replacing
Dict and Tuple usages in type annotations within this module (mistral.py) with
dict and tuple respectively; also ensure any annotations that referenced
Callable (e.g., callback or handler parameters) use the newly imported
collections.abc.Callable and that Any remains from typing if still needed.

In `@packages/ai/src/ai/common/chat.py`:
- Around line 508-515: The except block is re-raising mapped exceptions without
preserving the original cause, losing the original traceback; update both
re-raise sites in the except block (the two occurrences of raise
self.map_exception(e)) to chain the original exception (use "raise
self.map_exception(e) from e") so the original SDK/provider traceback is
preserved; locate this logic within the chat streaming method where variables
emitted, is_retryable, attempt, and max_network_retries are used and ensure
map_exception(e) is raised with "from e" in both branches.
- Around line 499-507: The closure gated defined inside the retry loop captures
the mutable emitted dict by reference which can lead to late-binding bugs across
iterations; modify the gated definition in the loop (where emitted is created
and gated is defined) to bind the current emitted dict as a default parameter
(e.g., def gated(text: str, _emitted=emitted): ...) so gated reads/writes the
iteration-specific container, ensuring _chat_stream/on_chunk use the correct
emitted state across retries.
- Line 16: Replace deprecated typing aliases: remove Callable and Dict from the
typing import and instead import Callable from collections.abc (and use the
built-in dict type). Update all occurrences of Dict[str, Any] to dict[str, Any]
and any typing.Callable uses to collections.abc.Callable (or just Callable after
changing the import) in this file (packages/ai/src/ai/common/chat.py) so
annotations use the modern Python 3.10+ forms; adjust the from ... import line
and all annotations referencing Dict or Callable accordingly.

---

Outside diff comments:
In `@nodes/src/nodes/llm_ollama/ollama.py`:
- Around line 57-60: The code calls serverbase.rstrip(...) without guarding
against serverbase being None (from config.get('serverbase', None)); update the
logic in ollama.py to check that serverbase is truthy before calling rstrip
(e.g., use "if serverbase and not serverbase.rstrip('/').endswith('/v1'):"),
leaving serverbase unchanged when None or empty, so you avoid AttributeError
while preserving the existing normalization behavior.

In `@nodes/src/nodes/llm_xai/xai.py`:
- Around line 60-62: The constructor calls the base class initializer twice
which re-runs ChatBase setup and can overwrite state; remove the duplicate
super().__init__(provider, connConfig, bag) so ChatBase is only initialized once
(keep the original call where the base is first initialized and delete the later
redundant call in the XAI constructor that re-invokes super().__init__).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 64e5c47e-1cab-4e53-ae5c-af5b9ab02885

📥 Commits

Reviewing files that changed from the base of the PR and between 5c88da5 and 10a30a9.

📒 Files selected for processing (16)
  • nodes/src/nodes/llm_anthropic/anthropic.py
  • nodes/src/nodes/llm_base/IInstance.py
  • nodes/src/nodes/llm_bedrock/bedrock.py
  • nodes/src/nodes/llm_deepseek/deepseek.py
  • nodes/src/nodes/llm_gemini/gemini.py
  • nodes/src/nodes/llm_gmi_cloud/gmi_cloud.py
  • nodes/src/nodes/llm_ibm_watson/ibm_watson.py
  • nodes/src/nodes/llm_mistral/mistral.py
  • nodes/src/nodes/llm_ollama/ollama.py
  • nodes/src/nodes/llm_openai/openai_client.py
  • nodes/src/nodes/llm_openai_api/openai_client.py
  • nodes/src/nodes/llm_perplexity/perplexity.py
  • nodes/src/nodes/llm_qwen/qwen_client.py
  • nodes/src/nodes/llm_vertex/vertex.py
  • nodes/src/nodes/llm_xai/xai.py
  • packages/ai/src/ai/common/chat.py

Comment thread nodes/src/nodes/llm_base/IInstance.py Resolved
Comment thread nodes/src/nodes/llm_base/IInstance.py Resolved
# =============================================================================

from typing import Any, Dict
from typing import Any, Callable, Dict

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Replace deprecated typing aliases with Python 3.10+ built-ins

Callable (newly added by this PR) should be imported from collections.abc; Dict should use the built-in dict. Both are flagged UP035 by Ruff.

As per coding guidelines, nodes/**/*.py targets Python 3.10+ and enforces Ruff.

♻️ Proposed fix
-from typing import Any, Callable, Dict
+from collections.abc import Callable
+from typing import Any

Update annotations:

-def __init__(self, provider: str, connConfig: Dict[str, Any], bag: Dict[str, Any]):
+def __init__(self, provider: str, connConfig: dict[str, Any], bag: dict[str, Any]):

-def _chat_stream(self, prompt: str, on_chunk: Callable[[str], None]) -> str:
+def _chat_stream(self, prompt: str, on_chunk: Callable[[str], None]) -> str:
🧰 Tools
🪛 Ruff (0.15.12)

[warning] 24-24: Import from collections.abc instead: Callable

Import from collections.abc

(UP035)


[warning] 24-24: typing.Dict is deprecated, use dict instead

(UP035)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/llm_gemini/gemini.py` at line 24, Replace deprecated typing
aliases: change the import line to only import Any from typing and import
Callable from collections.abc (e.g., "from typing import Any" and "from
collections.abc import Callable"), then update all type annotations that use
Dict to use the built-in dict (e.g., dict[str, Any] or dict[Any, Any] as
appropriate); ensure no other uses of typing.Callable/typing.Dict remain in
gemini.py.


import re
from typing import Any, Dict
from typing import Any, Callable, Dict

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Replace deprecated typing aliases with Python 3.10+ built-ins

Callable (newly added by this PR) should be imported from collections.abc; Dict should use the built-in dict. Both are flagged UP035 by Ruff.

As per coding guidelines, nodes/**/*.py targets Python 3.10+ and enforces Ruff.

♻️ Proposed fix
-from typing import Any, Callable, Dict
+from collections.abc import Callable
+from typing import Any

Update annotations:

-def __init__(self, provider: str, connConfig: Dict[str, Any], bag: Dict[str, Any]):
+def __init__(self, provider: str, connConfig: dict[str, Any], bag: dict[str, Any]):

-def _chat_stream(self, prompt: str, on_chunk: Callable[[str], None]) -> str:
+def _chat_stream(self, prompt: str, on_chunk: Callable[[str], None]) -> str:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
from typing import Any, Callable, Dict
from collections.abc import Callable
from typing import Any
🧰 Tools
🪛 Ruff (0.15.12)

[warning] 25-25: Import from collections.abc instead: Callable

Import from collections.abc

(UP035)


[warning] 25-25: typing.Dict is deprecated, use dict instead

(UP035)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/llm_ibm_watson/ibm_watson.py` at line 25, Replace deprecated
typing aliases in the module import: stop importing Callable and Dict from
typing; import Callable from collections.abc and use the built-in dict type for
annotations instead of typing.Dict. Update the import line in ibm_watson.py to
keep Any from typing (if needed) and ensure all function/type annotations that
used Dict[...] are changed to dict[...], and annotations using Callable use
collections.abc.Callable.

depends(requirements)

from typing import Any, Dict, Tuple
from typing import Any, Callable, Dict, Tuple

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Replace deprecated typing aliases with Python 3.10+ built-ins

Callable should be imported from collections.abc, and Dict/Tuple should use the built-in dict/tuple — all three are flagged UP035 by Ruff on this line. Callable and Tuple are newly added by this PR.

As per coding guidelines, nodes/**/*.py targets Python 3.10+ and enforces Ruff.

♻️ Proposed fix
-from typing import Any, Callable, Dict, Tuple
+from collections.abc import Callable
+from typing import Any

Then update all affected type annotations in this file:

-def __init__(self, provider: str, connConfig: Dict[str, Any], bag: Dict[str, Any]):
+def __init__(self, provider: str, connConfig: dict[str, Any], bag: dict[str, Any]):

-def _getRetryConfig(self, model: str) -> Tuple[int, float]:
+def _getRetryConfig(self, model: str) -> tuple[int, float]:

-def _chat_stream(self, prompt: str, on_chunk: Callable[[str], None]) -> str:
+def _chat_stream(self, prompt: str, on_chunk: Callable[[str], None]) -> str:
🧰 Tools
🪛 Ruff (0.15.12)

[warning] 40-40: Import from collections.abc instead: Callable

Import from collections.abc

(UP035)


[warning] 40-40: typing.Dict is deprecated, use dict instead

(UP035)


[warning] 40-40: typing.Tuple is deprecated, use tuple instead

(UP035)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/llm_mistral/mistral.py` at line 40, Replace deprecated typing
aliases on the import line by importing Callable from collections.abc and using
built-in dict and tuple types: update the import statement that currently brings
in Any, Callable, Dict, Tuple from typing, and change Callable to be imported
from collections.abc while replacing Dict and Tuple usages in type annotations
within this module (mistral.py) with dict and tuple respectively; also ensure
any annotations that referenced Callable (e.g., callback or handler parameters)
use the newly imported collections.abc.Callable and that Any remains from typing
if still needed.

import json
import importlib
from typing import Dict, Any
from typing import Dict, Any, Callable

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Replace deprecated typing aliases (Ruff UP035)

Callable and Dict are imported from typing. For Python 3.10+ (the project's declared target), Callable should come from collections.abc and Dict should be the built-in dict.

♻️ Proposed fix
-from typing import Dict, Any, Callable
+from collections.abc import Callable
+from typing import Any

Update all Dict[str, Any] annotations in this file to dict[str, Any].

🧰 Tools
🪛 Ruff (0.15.12)

[warning] 16-16: Import from collections.abc instead: Callable

Import from collections.abc

(UP035)


[warning] 16-16: typing.Dict is deprecated, use dict instead

(UP035)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai/src/ai/common/chat.py` at line 16, Replace deprecated typing
aliases: remove Callable and Dict from the typing import and instead import
Callable from collections.abc (and use the built-in dict type). Update all
occurrences of Dict[str, Any] to dict[str, Any] and any typing.Callable uses to
collections.abc.Callable (or just Callable after changing the import) in this
file (packages/ai/src/ai/common/chat.py) so annotations use the modern Python
3.10+ forms; adjust the from ... import line and all annotations referencing
Dict or Callable accordingly.

Comment on lines +499 to +507
for attempt in range(max_network_retries):
emitted = {'value': False}

def gated(text: str) -> None:
emitted['value'] = True
on_chunk(text)

try:
return self._chat_stream(prompt, gated)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

gated closure captures emitted by reference — Ruff B023

gated is re-defined each loop iteration and closes over emitted via late binding. If the loop executes a second iteration before any outstanding gated reference is consumed (e.g., if _chat_stream were ever adapted to an async/generator pattern), gated from iteration N would read emitted from iteration N+1, silently defeating the "did we emit?" gate. Ruff B023 flags this directly.

The fix is to bind the current iteration's dict at function-definition time via a default argument:

♻️ Proposed fix
-            def gated(text: str) -> None:
-                emitted['value'] = True
-                on_chunk(text)
+            def gated(text: str, _emitted: dict = emitted) -> None:
+                _emitted['value'] = True
+                on_chunk(text)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for attempt in range(max_network_retries):
emitted = {'value': False}
def gated(text: str) -> None:
emitted['value'] = True
on_chunk(text)
try:
return self._chat_stream(prompt, gated)
for attempt in range(max_network_retries):
emitted = {'value': False}
def gated(text: str, _emitted: dict = emitted) -> None:
_emitted['value'] = True
on_chunk(text)
try:
return self._chat_stream(prompt, gated)
🧰 Tools
🪛 Ruff (0.15.12)

[warning] 503-503: Function definition does not bind loop variable emitted

(B023)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai/src/ai/common/chat.py` around lines 499 - 507, The closure gated
defined inside the retry loop captures the mutable emitted dict by reference
which can lead to late-binding bugs across iterations; modify the gated
definition in the loop (where emitted is created and gated is defined) to bind
the current emitted dict as a default parameter (e.g., def gated(text: str,
_emitted=emitted): ...) so gated reads/writes the iteration-specific container,
ensuring _chat_stream/on_chunk use the correct emitted state across retries.

Comment thread packages/ai/src/ai/common/chat.py Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/ai/src/ai/common/chat.py`:
- Around line 550-555: The streaming path currently wraps getTokens(response) in
a silent try/except which hides tokenizer errors and diverges from chat_string;
change that to catch Exception as e and log the failure (e.g. call debug with a
clear message including e) instead of swallowing it, so failures in
getTokens(response) for the streaming branch are visible; keep the existing
logic using prompt_tokens, result_tokens and self._modelTotalTokens but replace
the bare except with a narrowed catch that logs the error (referencing
getTokens, response, result_tokens, prompt_tokens and self._modelTotalTokens).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 37735c0f-b0e6-4306-815f-8454d119c9c1

📥 Commits

Reviewing files that changed from the base of the PR and between 10a30a9 and 44f0181.

📒 Files selected for processing (2)
  • nodes/src/nodes/llm_base/IInstance.py
  • packages/ai/src/ai/common/chat.py

Comment thread packages/ai/src/ai/common/chat.py Resolved
@Rod-Christensen
Rod-Christensen marked this pull request as draft May 15, 2026 18:00
@charliegillet

Copy link
Copy Markdown
Contributor

Thanks @AditM99 for pursuing incremental LLM output to the UI. This draft targets the former llm_base/IInstance.py architecture and has been superseded by the current callback-based ChatBase/LLMBase streaming path and PR #1217. We are closing this implementation as superseded. If visible answer-token streaming remains incomplete, that product gap should be tracked in a fresh issue or PR against the current architecture, with your earlier contribution credited.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ai AI/ML modules module:nodes Python pipeline nodes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream LLM output incrementally to the UI

2 participants