Skip to content

fix(tool_slack): keep subtype and bot_id so channel_history records can be classified - #1827

Merged
dylan-savage merged 3 commits into
rocketride-org:developfrom
Aryan-070:fix/RR-1826-channel-history-discriminators
Sep 8, 2026
Merged

dylan-savage merged 3 commits into
rocketride-org:developfrom
Aryan-070:fix/RR-1826-channel-history-discriminators

Conversation

@Aryan-070

@Aryan-070 Aryan-070 commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • channel_history now returns subtype and bot_id, the two fields that classify a record, so system events can be filtered and an agent can skip its own posts.
  • Both are additive and absent on ordinary user messages, so the shape is unchanged for existing callers — pinned by a test.

Type

fix

conversations.history interleaves system events with real messages, and subtype is the only thing separating them. A channel_join entry carries a ts, a user, and a text of "<@U0123ABCD> has joined the channel" — after _clean_message it was byte-for-byte the shape of a real message. Same for channel_leave, channel_topic, channel_purpose, channel_name, channel_archive.

There was no workaround, because the field was discarded before the caller saw anything. Pattern-matching on text for "has joined the channel" is localisation-dependent and cannot distinguish a bot's post from a person's at all.

Measured: in one channel read, 30 records came back for 19 actual messages — 37% of what an agent was handed as conversation history was join/leave noise, complete with plausible timestamps and user ids. Since tool_slack is an agent tool (#1574), those records land in a model's context as evidence.

bot_id was dropped the same way, and the asymmetry there is the sharp bit: check_connection already returns the caller's own bot_id (slack_client.py:286, documented at README.md:47). So the node handed you a value to compare against and nothing to compare it to. The consequence is a feedback loop — an agent posting into a channel it also reads sees its own output as new input on the next run. Bot messages often carry no user at all, so user cannot filter them either.

Both fields are already present in the raw Slack payload; they were being deleted on the way out. The allowlist approach in _clean_message is right — the fix is not "return everything" — it just needs to include the fields that classify a record, not only the ones that carry its content.

Testing

  • Tests added or updated
  • Tested locally
  • ./builder test passes

Three tests added to the existing TestChannelHistory, beside test_thread_ts_included_when_present which they mirror. Full suite:

test/tool_slack/test_slack.py ..........................................
.........................................................        99 passed

96 of those are pre-existing and unchanged. The two behaviour tests were verified against pristine origin/develop:

test_subtype_included_so_system_events_can_be_filtered   FAILED
  assert result[1]['subtype'] == 'channel_join'
test_bot_id_included_so_an_agent_can_skip_its_own_posts  FAILED
  assert result[0]['bot_id'] == 'B-123'
test_ordinary_messages_are_unchanged                     PASSED
                                                         2 failed, 1 passed

test_ordinary_messages_are_unchanged passing in both directions is the point of it — it is the compatibility guard, asserting a plain user message still returns exactly {ts, user, text} and that unrelated Slack fields (team, blocks) are still dropped by the allowlist.

ruff check and ruff format --check pass.

./builder nodes:test was not run in full — it needs a built engine and a running task server, which I don't have provisioned. These tests need neither (they drive a mocked WebClient), so CI picks them up unchanged.

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included
  • Wiki updated (if applicable)
  • Breaking changes documented (if applicable)

nodes/src/nodes/tool_slack/README.md:50 is updated in the same change, since the returned-field list is the tool's public contract. The edit is a prose table row at :50; the ROCKETRIDE:GENERATED:PARAMS block at :141-158 is untouched.

Not a breaking change: both keys are additive and only appear on records that previously carried no marker at all.

Follow-up worth considering separately, not assumed here: whether other tool_* nodes drop a discriminator the same way.

Linked Issue

Fixes #1826

Summary by CodeRabbit

  • Enhancements

    • Slack channel history now preserves message subtype and bot identifier details, helping distinguish system events, bot-authored posts, and ordinary user messages.
    • Unrelated message fields continue to be excluded for a cleaner, consistent history response.
  • Documentation

    • Updated channel history guidance explains the available metadata and how to use it for filtering and message identification.
  • Tests

    • Added coverage confirming metadata is retained when present while ordinary messages maintain their existing format.

…an be classified

conversations.history interleaves system events with real messages, and
subtype is the only field that separates them. _clean_message stripped it,
so a channel_join -- which carries a ts, a user, and a text of
"<@U0123ABCD> has joined the channel" -- arrived byte-for-byte in the shape
of a real message, with no way to filter it and no way to recover the
distinction. In one channel read, 30 records came back for 19 actual
messages: 37% of what an agent was handed as conversation history was
join/leave noise with plausible timestamps and user ids.

bot_id was dropped the same way. check_connection already returns the
caller's own bot_id, so there was a value to compare against and nothing to
compare it to -- and an agent posting into a channel it also reads sees its
own output as new input on the next run. Bot messages often carry no user
at all, so `user` cannot filter them either.

Both fields are already in the raw payload; they were being deleted on the
way out. Both are absent on ordinary user messages, so the shape is
unchanged for existing callers -- pinned by a test.

The allowlist approach is right; it just needs to include the fields that
classify a record, not only the ones that carry its content.

README updated: the returned-field list is the tool's public contract.

Refs rocketride-org#1826
@github-actions github-actions Bot added docs Documentation module:nodes Python pipeline nodes labels Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: b0c2d183-76e6-4b8c-b611-9493ae5baaf9

📥 Commits

Reviewing files that changed from the base of the PR and between 5276306 and f9159c3.

📒 Files selected for processing (4)
  • nodes/src/nodes/tool_slack/IInstance.py
  • nodes/src/nodes/tool_slack/README.md
  • nodes/src/nodes/tool_slack/slack_client.py
  • nodes/test/tool_slack/test_slack.py

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

channel_history now preserves Slack subtype and bot_id fields when present. Tests cover system events, bot-authored messages, and unchanged ordinary message output. The README documents these optional fields and their filtering use.

Changes

Slack history field preservation

Layer / File(s) Summary
Preserve message classifiers
nodes/src/nodes/tool_slack/slack_client.py, nodes/src/nodes/tool_slack/IInstance.py, nodes/src/nodes/tool_slack/README.md, nodes/test/tool_slack/test_slack.py
_clean_message retains optional subtype and bot_id fields. Tool descriptions document their use for filtering system events and bot-authored messages. Tests cover system events, bot messages, file-share messages, and ordinary message output.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to f9159

Channel history now exposes optional Slack subtype and bot ID classifiers while keeping ordinary message output unchanged. The implementation, documentation, and compatibility coverage align, with no current merge-blocking risk identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 70.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 3 files. (1 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: preserving Slack subtype and bot_id fields for channel_history classification.
Linked Issues check ✅ Passed The changes satisfy issue #1826. channel_history preserves subtype and bot_id only when present, retains allowlist cleanup, preserves ordinary-message output, updates documentation, and adds tests for…
Out of Scope Changes check ✅ Passed The code, tests, and documentation changes directly support issue #1826. No unrelated or out-of-scope changes are identified.
Full details: Docstring Coverage

Explanation

Docstring coverage is 70.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 3 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@nodes/src/nodes/tool_slack/README.md`:
- Line 50: Update the channel_history documentation to remove bot_message from
the system-event subtype examples; keep channel_join and channel_topic as
system/event examples, and describe bot_message separately as identifying
bot-authored messages without changing the bot_id guidance.

In `@nodes/src/nodes/tool_slack/slack_client.py`:
- Around line 197-220: Update the channel_history Returns documentation to
include the optional subtype and bot_id fields emitted by _clean_message,
alongside the existing ts, user, text, and thread_ts fields. Keep the documented
response structure and field descriptions consistent with the runtime behavior
and README.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 675c9f61-1ce8-4585-9c01-40ecd9eebff5

📥 Commits

Reviewing files that changed from the base of the PR and between 243c1f2 and 7149c77.

📒 Files selected for processing (3)
  • nodes/src/nodes/tool_slack/README.md
  • nodes/src/nodes/tool_slack/slack_client.py
  • nodes/test/tool_slack/test_slack.py

Comment thread nodes/src/nodes/tool_slack/README.md Resolved
Per CodeRabbit, two things.

channel_history's own Returns block still described only ts/user/text and
thread_ts. _clean_message and the README were updated with the new fields but
the method docstring was not, which is the one a caller reads first.

The README also listed `bot_message` among the system-event subtype examples.
It is not a system event in the channel_join sense -- it classifies a
bot-authored post. Now described separately from the join/leave/topic events,
alongside the bot_id guidance it actually relates to.

Refs rocketride-org#1826
@Aryan-070

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nihalnihalani

Copy link
Copy Markdown
Collaborator

Checked this against develop and found nothing blocking. _clean_message still returns only ts/user/text (+thread_ts), so the bug is live; the only change to tool_slack since it landed is #2008, which touched message_post, so this does not collide with it. The README edit sits in the hand-written tools table above the ROCKETRIDE:GENERATED:PARAMS block, so nodes:docs-generate won't fight it, and CI OK is green.

It has been sitting since 2026-08-05 with no human review — worth pinging for one.

… it to the agent

The tool description still promised only ts, user and text, so the new
discriminators never reached the LLM. It now carries the contract.

The guidance was also wrong: treating any subtype as noise drops real
messages (file_share, thread_broadcast, me_message), and bot_message is
absent when an app posts under its own bot identity, so bot_id is the
reliable discriminator. Adds tests for both shapes.
@dylan-savage
dylan-savage merged commit 9429cbc into rocketride-org:develop Sep 8, 2026
24 checks passed
dk-rocketride added a commit that referenced this pull request Nov 4, 2026
* feat(llm_openai): make temperature configurable (#1709)

* feat(llm_openai): make temperature configurable

temperature was hardcoded to 0 when constructing ChatOpenAI for
non-reasoning models, with no way to override it via the node's config.
This forced fully deterministic output regardless of use case, with no
path for creative writing, brainstorming, or varied-response pipelines
short of editing the node's source directly.

Read temperature via config.get('temperature', 0) and pass it through
to ChatOpenAI(...) instead of the hardcoded 0, following the existing
pattern already working in llm_ollama. Defaults to the current
behavior when unset, so it's fully backward-compatible. Exposed for
every OpenAI profile (not just custom), since the motivating use cases
apply to named models like gpt-4o/gpt-5.2 as much as custom ones.

Closes #1660

* fix(llm_openai): address review feedback on temperature field

- Bound temperature to [0, 2] (OpenAI's accepted range) so out-of-range
  values fail config validation before reaching the client.
- Soften "deterministic" claims to reflect that temperature 0 does not
  guarantee identical OpenAI output for identical input.

* docs(llm_openai): document temperature's 0-2 range in README

services.json bounds temperature to [0, 2]; the README field table
didn't state that range.

* docs(llm_openai): regenerate schema block for the temperature field

Run via nodes:docs-generate (with a temporary branch rename to satisfy
its release-track-branch gate, then reverted) rather than hand-editing
the generated block, per the repo's doc-generation convention.

* fix(llm_openai): declare temperature's default in the schema

Add "default": 0 to services.json so the generated Schema table's
Default column isn't blank, and drop the now-redundant "Defaults to
0." prose from the description (the default field carries it).
Regenerated via nodes:docs-generate (temporary branch rename to
satisfy its release-track gate, reverted after), not hand-edited.

* fix(llm_openai): nest temperature under profile objects so it reaches ChatOpenAI

temperature lived in the top-level shape[] section, which config
merging treats as connConfig root, so Config.getNodeConfig's
profile branch never saw it — the field rendered in the UI but
never affected the request. Move it into all openai.* profile
objects (matching the llm_ollama pattern) and swap the .get(...,
0) fallback for an 'in' check so a present null isn't silently
overridden.

Adds nodes/test/llm_openai/test_temperature.py to cover the
plumbing, since the existing declarative tests only assert on
response text.

---------

Co-authored-by: madhumithachandrasekaran31 <madhumitha2110549@ssn.edu.in>

* chore(models): sync LLM model lists (#1784)

Co-authored-by: asclearuc <29624014+asclearuc@users.noreply.github.com>

* feat(nodes): add text Chunker pipeline node (#897)

* feat(nodes): add text Chunker pipeline node

Adds a `chunker` pipeline node that splits long documents into smaller
chunks suitable for embedding and retrieval. Multiple chunking strategies
are bundled: recursive character splitting, sentence-aware splitting, and
token-based splitting.

Components:
- IGlobal — standard lifecycle with depends() for SDK auto-install.
- IInstance — receives documents, applies the configured strategy, writes
  chunked documents to the documents output lane.
- chunker_strategies — pluggable strategy implementations.
- services.json — strategy profiles with configurable chunk_size and overlap.

Resubmission of the chunker portion of closed PR #499 per maintainer
feedback: one PR per node, clean branch from develop, no unrelated
file changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(chunker): mark node experimental and tighten field types

Set capabilities to ["experimental"] (required for new nodes) and switch
chunk_size/chunk_overlap from "number" to "integer" with minimum bounds.
Cross-field validation (overlap < size) remains enforced at runtime in
IGlobal.beginGlobal and in each strategy constructor.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(chunker): raise on unknown strategy instead of silently defaulting

Add an explicit elif branch for 'recursive' and raise ValueError when the
configured strategy_name is none of recursive/sentence/token. Previously a
typo (e.g. 'recurisve') would silently fall through to recursive, masking
bad config in production.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(chunker): drop dead pending-docs buffer and always preventDefault

writeDocuments() emits chunks synchronously and never populated the
_pending_docs buffer, so _flush_documents()/closing()/close() were dead
helpers that risked re-emitting if anything ever did populate the buffer.
Remove them; keep open() solely to reset chunkId per object.

Also always call preventDefault() after handling a batch: previously, if
every input document produced zero chunks (empty/whitespace), the
originals leaked downstream.

Switch DocMetadata construction to model_validate() when available; this
avoids colliding with DocMetadata.__init__'s positional ``pInstance`` arg
and matches the pattern already used for Doc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(chunker): preserve separators, cap overlap, and safe tiktoken decode

Three correctness fixes in chunker_strategies:

1. RecursiveCharacterChunker._split_text: when a separator-driven emit
   happens, append the separator to the emitted chunk instead of dropping
   it. Without this, every overflow silently lost separator characters,
   causing chunk reconstruction to diverge from the source text.

2. RecursiveCharacterChunker.chunk: cap the prepended overlap to the
   remaining budget (chunk_size - len(raw)) so an already-full chunk plus
   overlap can never exceed chunk_size. Use the *applied* overlap when
   adjusting start_char so offsets stay truthful.

3. TokenChunker: tiktoken's Encoding.decode does not accept an ``errors=``
   kwarg (passing one raises TypeError). Replace the bogus fallback with
   _safe_decode, which falls back to decode_single_token_bytes per token
   and uses errors='replace' at the bytes -> str boundary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(chunker): relocate suite to nodes/test/chunker/test_all.py

Match the per-node test convention used by agent_rocketride,
tool_exa_search and local_text_output: a ``test_all.py`` under
``nodes/test/<node>/`` rather than the legacy ``test/nodes/test_*.py``
path.

Drop the sys.modules stubs for rocketlib/ai.common.schema/depends
(maintainer policy: do not mock built-in interpreter modules) and replace
the ``spec_from_file_location`` path imports with normal package imports
(``from chunker.chunker_strategies import ...``) after putting
``nodes/src/nodes`` on sys.path.

The package import is guarded by ``pytest.importorskip('chunker')`` so a
vanilla local pytest run (without the build interpreter that provides
rocketlib) skips cleanly instead of erroring at collection.

Coverage retained: every assertion from the original suite plus three new
regression tests for the fixes in this PR — separator preservation on
overflow, overlap capping under chunk_size, preventDefault on empty
batches, ValueError on unknown strategy_name, and _safe_decode fallback.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(chunker): assert max decode input <= chunk_size in token guard

Strengthen the O(n^2) decode regression guard: track decode input token
lengths in the mock encoder and assert max(decoded_lengths) <= chunk_size,
in addition to the existing linear call-count assertion. A growing-prefix
decode regression can keep the call count linear while inflating individual
decode inputs, which the count-only check would miss.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(chunker): drop vanilla-pytest fallbacks; rely on build interpreter

Per maintainer review: the build interpreter is mandatory and configures
import paths. Remove the sys.path manipulation, pytest.importorskip, and the
try/except pytest.skip guards in _import_node_classes()/_import_schema() so a
missing module surfaces as a collection error instead of a silent skip.

* test(chunker): remove skip fallbacks but keep required sys.path

Removing pytest.importorskip and the try/except pytest.skip guards (per review)
so missing build-interpreter modules surface as collection errors instead of
silent skips. Restored the nodes/src/nodes sys.path prefix: without it the node
package is not importable even under the build interpreter (CI: ModuleNotFound
'chunker'), matching every other node suite (local_text_output, milvus, ...).

* fix(nodes): stop test dirs from shadowing node packages on sys.path

The chunker suite's addition perturbs pytest's collection order, which
exposed a latent fragility in the shared "prepend nodes/src/nodes" import
pattern: `nodes/test/<node>/` is itself importable as `<node>` (it has an
`__init__.py`), and a plain `if str(NODES_SRC) not in sys.path` guard does
not guarantee the node source is searched *before* the test directory. When
another node suite had already appended `nodes/src/nodes` behind the test
dir, `import local_text_output` resolved to `nodes/test/local_text_output`
(which has no `IInstance`), failing collection with
`ModuleNotFoundError: No module named 'local_text_output.IInstance'` on
Ubuntu and macOS.

Force `nodes/src/nodes` to the front of `sys.path` in both the chunker and
local_text_output suites so the real node package always wins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(nodes): address chunker review — icon, README, overlap bug, classType (#1643)

Review response (joshuadarron, CHANGES_REQUESTED):
- Add co-located chunker.svg icon (services.json referenced a missing
  asset — broken node-picker render).
- Add co-located README.md with the generated schema/dependencies/source
  block, config fields, and lanes (co-location rule; 106/120 nodes ship
  one).
- Fix overlap silently dropping to 0 on full-size chunks: split now
  reserves overlap headroom (_split_size = chunk_size - overlap) and
  reads the overlap from the original text preceding each raw chunk, so
  overlap is honored on the common prose/hard-split path. Cap the applied
  overlap by min(chunk_overlap, raw_start, chunk_size - len(raw)) so a
  trailing separator on a full raw chunk can never push an emitted chunk
  past chunk_size — the two regression tests (overlap honored on a
  full-size chunk; no chunk exceeds chunk_size with real '. ' separators,
  input found by fuzzing) fail on the pre-fix code and pass after.
- classType 'transform' -> 'preprocessor' (the established category for
  text splitters; no other node uses 'transform').

Closes #1643

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(nodes): chunker — enforce size cap for overlap<separator, drop global sys.path mutation

Follow-up to review response, from a fresh-eyes audit of my own fix:

- Real invariant bug: the previous cap only trimmed the overlap, so a raw
  chunk emitted as current+separator could still exceed chunk_size when
  chunk_overlap was smaller than the trailing separator (e.g. default
  '\n\n' with overlap 0 or 1 — both schema-allowed). Cap the emitted body
  at chunk_size (dropping only excess trailing separator chars, never
  content); current always fits _split_size so no content is lost. 0
  invariant violations across 200k fuzz trials with random multi-char
  separators; full overlap still honored on true full-size chunks.
  Regression test added for overlap 0/1 with default separators.
- Test import: load chunker_strategies.py directly (pure stdlib) instead
  of importing the chunker package and permanently prepending src/nodes to
  sys.path at module import; the IGlobal/IInstance tests scope that
  insertion to a restored context manager. Removes a global import-state
  side effect that could perturb how sibling node suites resolve
  same-named test-vs-source packages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(chunker): revert import rework, keep the size-cap fix + regression tests

My previous commit's direct-file load of chunker_strategies.py made the
module-level strategy classes distinct objects from the ones the chunker
package instantiates, so the IGlobal lifecycle tests'
isinstance(strategy, SentenceChunker) failed. Restore the working package
import (as on 908d3c66, whose chunker tests passed) and keep the three
overlap invariant tests. The chunker_strategies.py emit_len size-cap fix
is unchanged.

Note: the separate image_cleanup collection failure is a pre-existing
fragility in upstream test/image_cleanup/test_provenance.py (it assumes
'import image_cleanup' resolves to the source node; the same-named test
package can shadow it under xdist), surfaced by adding a node test — not
a chunker defect. A hardening fix for that test is proposed separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(chunker): add documentation link and refresh stale suite names

Two small alignments with today's develop:

- services.json gained the `documentation` key that 141 of develop's
  services.json files already carry (bare docs root, placed right after
  `icon`). Without it the node's Documentation menu entry has no target.
- The test module docstring still referred to the `milvus` and `pinecone`
  suites by their pre-ddf94dd6 names; they are `store_milvus` and
  `store_pinecone` on develop. Comment-only change, no behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(nodes): add chunker to the node catalog and changelog (#1643)

Every node PR merged since July adds a catalog row and a changelog entry;
this PR was written before that became the norm.

- docs/README-nodes.md: new Document Processing row for `chunker`
  (documents -> documents). The header counts on lines 13-14 are already
  stale against develop, so they are deliberately left alone rather than
  bumped by one on a number that is wrong either way.
- CHANGELOG.md: [Unreleased] / ### Added line, alphabetical among the
  existing **nodes** entries.

Note: open PR #2139 removes docs/README-nodes.md as part of the docs
revamp. If that lands first this row goes away with the file; nothing
else in this PR depends on it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(chunker): add the MIT header to the node package __init__

CONTRIBUTING.md requires the MIT header block on Python sources, and every
other file in this node (IGlobal.py, IInstance.py, chunker_strategies.py)
already carries it — only the package __init__.py was missing it. Copied
verbatim from the canonical form used by 94 of develop's node __init__.py
files (e.g. nodes/src/nodes/__init__.py); byte-identical to the block on
this node's own IGlobal.py.

nodes/test/chunker/__init__.py is deliberately left empty: only 23 of
develop's 83 node test-package __init__.py files carry a header, and 44 of
them are empty, so the majority convention is no header there.

Comment-only change: no behaviour, no imports, no exports touched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(chunker): drop recursive strategy, delegate to preprocessor_langchain

The preprocessor_langchain node already exposes LangChain's
RecursiveCharacterTextSplitter through its `default` and `recursive` profiles
(nodes/src/nodes/preprocessor_langchain/services.json, langchain.py:186-204),
so shipping a second recursive implementation gave the product two ways to do
the same thing and left pipeline authors to guess which node to reach for.

Narrow this node to the two strategies the engine does not already have:

- sentence: stdlib-regex sentence grouping, no third-party dependency and no
  language-model download (unlike the NLTK/Spacy profiles of General Text).
- token: real tiktoken BPE counts, versus the byte-length approximation
  (`bytes/3`) that General Text labels "Estimated tokens" (langchain.py:50-59).

`sentence` becomes the default profile. A stale `strategy: recursive` config
now raises at startup naming preprocessor_langchain rather than silently
falling back, so existing pipelines fail loudly with a pointer.

Docs record why the split exists and what each node is for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(chunker): document the cost of defaulting to the sentence strategy

A sentence is indivisible in this strategy, so chunk_size is a grouping target
rather than a hard cap. With 'sentence' now the default profile, input that
carries no sentence-ending punctuation (log lines, CSV rows, minified text,
OCR dumps) has no boundary to group on and comes back as a single oversized
chunk -- previously the recursive default would have hard-split it.

Document the case in the README, route those inputs to the token strategy
(which caps unconditionally) or the General Text node, and pin both halves of
the behavior in a test so the trade-off stays explicit.

No behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: nihalnihalani <24360630+nihalnihalani@users.noreply.github.com>

* feat(nodes): add Hybrid Rerank pipeline node (#899)

* feat(nodes): add search_hybrid BM25 + vector RRF re-ranker node

search_hybrid is a post-retrieval re-ranker: it takes the documents an
upstream vector store already attached to a question and re-orders them by
fusing the upstream vector `Doc.score` with a rank_bm25 BM25 score through
weighted Reciprocal Rank Fusion. It issues no fresh embedding lookup, so it
cannot surface documents the vector store did not return — hence classType
`rerank` (next to rerank_cohere) and capabilities `experimental`.

Ships the full node package (IGlobal/IInstance/hybrid_search, services.json
with balanced/semantic/keyword profiles exposing alpha/top_k/rrf_k, co-located
README with the generated-params markers, monochrome icon, requirements.txt)
plus nodes/test/search_hybrid/test_all.py covering the fusion engine and the
IInstance contract, and a services.json `test` block for the dynamic runner.

The rank_bm25 stand-in used when the real package is absent stays test-local
and find_spec-gated rather than living in nodes/test/mocks/, because that
directory is prepended to sys.path in every engine node subprocess via
ROCKETRIDE_MOCK and would shadow the real compute library during the dynamic
services.json test.

Recreated on top of develop as a single commit instead of replaying the
reviewed history: two of the old commits only added .gitleaksignore
fingerprints keyed to pre-rebase commit SHAs, which are dead after any rebase,
and a third only added the inline `# gitleaks:allow` on the BM25Okapi stub
assignment. Folding them in keeps every commit in the range clean under
gitleaks without touching .gitleaksignore at all.

Closes #619

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(nodes): catalog row and changelog entry for search_hybrid

Every node PR merged since July adds a row to the docs/README-nodes.md
catalog and a `- **nodes**: ...` line under CHANGELOG [Unreleased] / Added;
this PR was written before that became the norm.

The row goes under "Safety, Reranking & Search" next to rerank_cohere, which
shares the `rerank` classType. The stale "(121 node directories -> 151
services)" counts on line 14 are left untouched: develop is already at 137
directories / 183 services*.json, so a +1 bump would still be wrong.

Kept as a separate commit on purpose so it can be dropped without touching
the feature commit if PR #2139 (docs revamp), which deletes
docs/README-nodes.md outright, merges first.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(nodes): treat a missing document score as absent signal in search_hybrid

`Doc.score` is declared `score: float = Field(None, ...)`, so any upstream node
that does not score its output hands this node documents whose score is None.
IInstance coerced that to 0.0, and because sorting equal keys is stable the
"vector" list was really the documents in arrival order. RRF then fused that
order with weight alpha as though it were vector relevance, so the output looked
like a ranking while being partly just the order the documents came in — and it
failed quietly, because the results stayed plausible.

Distinguish absence of evidence from a real 0.0 ("the store scored this document
and it scored badly"):

- IInstance passes None through instead of 0.0.
- search() omits unscored documents from the vector-ranked list rather than
  inventing a 0.0 rank for them. RRF is rank-based, so a document absent from
  one list simply contributes nothing from that list; the document is still
  ranked via BM25.
- With no document scored, the vector list is empty and the existing BM25-only
  path takes over, matching `vector_scores=None`.
- IInstance warns once per instance when some or all documents arrive unscored,
  so the missing signal is visible to the pipeline author.

Also restores the deep copy in search() that the sibling methods already use;
it was fixed in 1da58b8 and lost when the branch was rebuilt.

Regression cover in nodes/test/search_hybrid/test_all.py: the headline test
feeds the same documents in two different arrival orders and asserts the ranking
is identical. Before the fix the two orders disagreed. Also covers all-unscored,
mixed scored/unscored, real-0.0-is-not-missing, and stable tie-handling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(nodes): name search_hybrid "Hybrid Rerank" and say what it is not

Two reviewers landed on the same framing objection: a node called "Hybrid
Search" reads as a promise to retrieve, and this node does not retrieve. It
re-ranks the candidate set an upstream vector store already returned, so a
keyword-relevant document the vector search missed can never be surfaced —
which is the main reason to reach for BM25 in the first place.

Running an independent keyword retrieval over the corpus is a different node,
not a review fix, so take the other option offered: make the name honest.

- services.json title/prefix become "Hybrid Rerank", matching the peer node
  "Cohere Rerank" and the `rerank` classType this node already declares.
- protocol, path and logicalType stay `search_hybrid`, so existing pipelines
  and the merged catalog rows are unaffected. This is a display-name change.
- README leads with the limitation rather than burying it, and documents the
  unscored-document handling introduced in the previous commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(nodes): cover the unscored-document path in search_hybrid's dynamic test

The existing services.json case feeds documents that all carry a score, so the
engine-level regression for unscored input was only exercised by the pytest
suite. Add a second case whose documents carry no score at all and whose
keyword-relevant document is deliberately last in arrival order, asserting it
comes back first.

Checked against nodes/test/framework/expectations.ExpectationValidator: the
expectation passes on the post-fix ordering and fails on the pre-fix ordering
(arrival order first), so the case genuinely discriminates rather than passing
either way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(nodes): stop search_hybrid's alpha endpoints dropping documents

a4d0e693 kept unscored documents out of the vector-ranked list, which is the
right reading of "no vector evidence" -- but `alpha == 1.0` returned that list
alone, so those documents stopped appearing in the output at all. Verified
counterexample: alpha=1.0 with two scored documents and one unscored returned
['a', 'c'] -- 'b' silently gone, where before a4d0e693 it was returned
mis-ranked. Trading mis-ranking for silent document loss is a bad trade, and
"results dropped, not passed through" was already a review concern on #899.
`alpha` is a bare number field with no min/max, so 1.0 is reachable.

Fixed by deleting the two endpoint short-circuits rather than special-casing
them further: alpha 0.0 and 1.0 now fall through to the same weighted RRF as
every other value, with weights [1.0, 0.0] and [0.0, 1.0]. Chosen over
appending the missing documents by hand because RRF already expresses exactly
the intended semantics -- a leg weighted 0.0 gives each of its documents a 0.0
contribution, so they sort below everything the weighted leg ranked while still
being returned -- and it needs no new ranking code, no second identity-matching
implementation, and no third notion of what an endpoint means.

It also removes the discontinuity the README itself flagged as a wart, which is
what the loss was made of: alpha=0.99 returned ['a', 'c', 'b'] and alpha=1.0
returned ['a', 'c'], so one hundredth of a config value deleted a document.
The two now agree exactly. Scored documents keep their pure vector ranking
(RRF over a single weighted list preserves that list's order); the unscored
ones follow in BM25 order.

Mirror-checked the other endpoint and the degenerate paths:

- alpha == 0.0 had the same defect reflected: a document carrying a real vector
  score whose text tokenizes to nothing is absent from the BM25 leg and was
  dropped. Same fix, same mechanism.
- `not vector_results` / `not bm25_results` keep their short-circuits. An empty
  leg holds no document, so returning the other leg unchanged cannot drop one;
  these also preserve the documented raw-BM25/raw-vector fallback score that
  TestMissingVectorScores pins.
- A document with no score *and* no BM25-rankable text is still not returned.
  It has no signal in either leg, and that is true at every alpha rather than
  only at an endpoint, so it is a property of the evidence, not of the mode.

Invariant now under test: the result set is exactly the documents with evidence
in at least one leg, at every alpha.

Tests: 64 -> 90. New TestAlphaEndpointsKeepEveryDocument covers the audit
counterexample, the alpha=0.0 mirror, permutation-invariance at both endpoints,
endpoint/neighbouring-blend agreement, and top_k. Verified they discriminate:
run against the node source at bac01b14 they fail, at both endpoints.

README: the missing-score section and the Downstream-consumer notes contradicted
each other (one said unscored documents "still appear in the output", the other
said alpha==1.0 returns the pure vector list). Both now describe the single new
behaviour, for all alpha; every claim was checked against the real rank_bm25.

Also renames the user-visible answer string to "Hybrid rerank returned ...",
matching the node's display title -- it re-orders an existing candidate set
rather than searching for one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: nihalnihalani <24360630+nihalnihalani@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(nodes/elasticsearch): decouple store mode from deployment profile (#1018)

* fix(nodes/elasticsearch): decouple store mode from deployment profile

Renamed elasticsearch.mode to elasticsearch.store_mode in services JSON to prevent the boolean checkbox value from overwriting the deployment profile string. Updated IGlobal.py to read store_mode, falling back to mode for legacy compatibility.

* test(nodes/elasticsearch): add PEP 257 docstrings and integration tests for store_mode fallback

- Consolidate two redundant _parse_mode_elasticsearch unit tests into one
  comprehensive test covering all input variants (bool, string aliases,
  unknown strings)
- Add PEP 257 triple-double-quoted docstrings to all test functions
- Add integration tests for IGlobal._begin_elasticsearch covering all 4
  fallback scenarios requested in CodeRabbit review:
  1. store_mode present: uses store_mode, ignores legacy mode
  2. Only legacy boolean mode: falls back to MODE_INDEX/MODE_VSTORE
  3. Only legacy deployment-profile string: maps correctly via fallback
  4. Both keys present: store_mode always wins
  5. Edge case: neither key -> defaults to MODE_VSTORE
- Patch Store constructor and beginGlobal to keep tests hermetic

Addresses: coderabbitai review comments on PR #1018

* test(store_elasticsearch): point the suite at the node's current name

The node was index_search when this branch was written and is store_elasticsearch
on develop, so every import here resolved to a module that no longer exists and
the suite failed at collection -- taking the three build jobs with it.

_begin_elasticsearch imports Store lazily, so patching that target needs the
submodule to exist. Seeding a stub in sys.modules satisfies the import without
loading the real elasticsearch client, which this suite has no reason to need.

Also drops an unused pytest import and the double quotes ruff rejected.

* docs(store_elasticsearch): name the renamed store_mode field

The field became elasticsearch.store_mode in services.json, so both the config
table and the generated params block still pointed at a key that no longer
exists. A straight rename, matching what nodes:docs-generate emits.

---------

Co-authored-by: aryaMehta26 <aryaMehta26@users.noreply.github.com>
Co-authored-by: dsapandora <ariel.vernaza@rocketride.ai>

* fix(store): honor _createCollection() return value before indexing (#1559)

* fix(store): honor _createCollection() return value before indexing

DocumentStoreBase.createCollection() discarded the bool returned by the
provider-specific _createCollection(), so a driver that reported failure by
returning False was ignored: the base proceeded to addChunks() against a
collection that was never created, crashing cryptically far from the real
cause.

createCollection() now captures the return value and raises a clear error when
it is exactly False, before calling addChunks(). `is False` is deliberate --
drivers that return None on success (store_weaviate, store_postgres,
rocketride_vector) are unaffected, and drivers that raise on failure
(store_qdrant, store_pinecone) propagate their own error before the check is
reached.

store_milvus._createCollection had inverted semantics (returned False on
success, True on failure), which honoring the return value would otherwise
break. Corrected to the documented contract: return True on success and let
real failures raise, removing a try/except that swallowed the real error.

Fixes #1495

* test(store): cover the None-success and error-swallowing driver shapes

Three gaps in the original coverage, all on the contract the `is False` check
depends on:

- None was untested, yet it is the reason the check is `is False` rather than
  a falsy test. store_weaviate, store_postgres and rocketride_vector create the
  collection and return nothing; a falsy check would break all three. Pin it.

- The False case asserted only that *something* raised. #1495 is specifically
  about the failure being named -- the old behaviour surfaced as a cryptic
  AttributeError from inside addChunks, far from the driver that failed -- so
  assert the message identifies the store.

- store_astra (nodes/src/nodes/store_astra/astra_db.py:143-145) catches every
  exception from create_collection, warns, and returns False. The real provider
  error never reaches the base, so `is False` is the only remaining signal that
  anything went wrong. That driver shape now has a regression test.

Also annotate the abstract _createCollection as bool | None. Implementations
may return True, return None, or raise; only an explicit False means failure,
and the signature said bool.

* ci(docker): start the image we just published, before signing it (#1847)

* ci(docker): start the image we just published, before signing it

We build, push and sign the engine image without ever running it. Three
broken images reached the registry that way — onnxruntime-gpu==1.20.1
unsatisfiable (#1754), the runtime install set, and No module named 'jwt'
(#1828) — and each was found by a user pulling latest, not by us.

Each of those exits within ~60s of docker run, so the check is cheap:
start the image by digest, fail immediately if the container dies, and
otherwise wait for /version. Placed before cosign so a container that
cannot start does not acquire a signature.

* ci(docker): require HTTP 200 (not merely non-000) from the smoke test

A 404/500 from a half-started engine was treated as success; only a real 200
now passes, and non-200 codes keep polling until the deadline. Addresses the
CodeRabbit Major finding on #1847.

---------

Co-authored-by: dk-rocketride <dmitrii.kataraev@rocketride.ai>

* fix(helm): require a shared file store before the chart will scale out (#1939)

* fix(helm): require a shared file store before the chart will scale out

The chart's README tells operators to run replicaCount 2 with autoscaling
for HA. Following that produces a deployment whose file store is private
to each pod.

RR_STORE_URL selects the account file-store backend; unset, Store.create()
falls back to filesystem://~/.rocketlib/dtc — a path inside the container.
The chart never set it, ships no PersistentVolume, and the Service has no
sessionAffinity, so with 2+ replicas a file written through one pod is
simply absent when the next request lands on another.

Signed downloads fail the same way and the engine already said so:
_ensure_signing_key() in ai/web/server.py mints an ephemeral per-process
RR_SIGNING_KEY on the filesystem backend, and its docstring warns the
result is "not valid across replicas — each process mints its own".

Nothing caught it. validateSecrets only asserts that some secret exists,
and tests/values_test.yaml — the fixture CI renders — is itself the broken
configuration: autoscaling to maxReplicas 5 with neither variable set.

validateScaleOut now fails the render when the effective replica count can
exceed 1 without RR_STORE_URL naming a shared backend (s3:// or
azureblob://), which fixes both halves at once — those presign natively, so
no signing key is involved. engine.sharedStoreConfigured is the escape
hatch for operators who inject the URL through existingSecret, which the
chart cannot read.

Also surfaces RR_STORE_URL, RR_STORE_SECRET_KEY and RR_SIGNING_KEY in
values.yaml, adds engine.service.sessionAffinity as a documented stopgap,
rewrites the HA Tuning section to state the precondition, and fixes the CI
fixture so it exercises a configuration that would actually work.

Durability at replicaCount 1 (no PVC, store lost on restart) is a separate
gap — a ReadWriteOnce PVC is itself incompatible with scale-out — and is
called out in the README rather than folded in here.

Closes #1938

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(helm): name the azure:// alias the scale-out guard already accepts

Review follow-up on #1939.

The guard accepts azure:// alongside s3:// and azureblob://, matching
store.py's dispatch (`elif scheme == 'azureblob' or scheme == 'azure'`), but
the failure message, values.yaml comment and README all listed only two of
the three. Reported as "the guard accepts an undocumented scheme"; the
inconsistency is real, but the fix belongs on the documentation side —
rejecting azure:// would make the chart refuse a configuration the engine
supports and that is genuinely shared.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: dk-rocketride <dmitrii.kataraev@rocketride.ai>

* chore(deps): refresh universal node constraints lockfile (#2212)

* perf(nodes): make anomaly detector z-score O(1) incremental (#1626)

* perf(nodes): make anomaly detector z-score O(1) incremental

The z_score method recomputed mean and standard deviation from scratch on
every value: it copied the whole window under the lock and ran a two-pass
scan over it. That made scoring O(window_size) per value, and with a window
configurable up to 10000 on a per-pipeline shared detector it was a real
throughput hotspot.

Maintain the mean and variance incrementally instead. The detector now keeps
assumed-mean shifted running sums over the window and adjusts them in O(1) as
values enter and leave, so scoring is O(1) amortized. A shift constant plus a
periodic full rebuild (once per window turnover) keeps the running sums
accurate on large-magnitude, low-spread streams where the naive
sum_sq/n - mean**2 identity loses precision to cancellation.

Output is unchanged: score, severity, is_anomalous and the details string all
match the previous two-pass result. The z_score path also no longer copies the
window under the lock, which shortens the critical section. IQR and rolling_avg
are untouched.

A new test file pins the incremental output to an independent two-pass oracle
(exact 4-decimal equality) across evictions, spikes, large magnitudes, the
recompute boundary, non-finite no-ops, and concurrent access. The existing
tests are unchanged and still pass.

* test(nodes): strengthen z-score large-magnitude checks and add PEP 257 docstrings

Address review on #1626:

- Large-magnitude streams (~1e9) now compare parsed mean, std, and z against
  the two-pass oracle within explicit tolerances instead of only asserting
  finiteness. At 1e9 four decimals are well within float64 resolution, so the
  previous sanity-only checks would have accepted arbitrarily inaccurate values.
- Concurrency consistency check no longer rebuilds the aggregates before the
  final comparison, so an update lost to a race stays observable.
- Add docstrings to every public test method and the module helpers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci: pin ruff version so an upstream release cannot redden develop (#1900)

* ci: pin ruff version so an upstream release cannot redden develop

* ci: pin ruff to 0.16.5 (freeze current version, not a downgrade)

Bumping the pin from 0.16.2 to 0.16.5 — the version CI has been running (latest)
— so pinning introduces no behavior change; 0.16.2 would be a downgrade whose
formatter could redden develop's already-0.16.5-formatted code.

---------

Co-authored-by: Dmitrii Karataev <dmitry.karataev@gmail.com>
Co-authored-by: dk-rocketride <dmitrii.kataraev@rocketride.ai>

* fix(store_weaviate): support Weaviate Cloud clusters that only allow the hfresh vector index (#1769)

* fix(store_weaviate): support Weaviate Cloud clusters that only allow the hfresh vector index

New-generation Weaviate Cloud clusters reject hnsw at collection
creation with 422 CONFIG_NOT_ALLOWED (allowed values: hfresh), so a
publish to such a cluster fails before inserting anything.

- retry collection creation with the hfresh vector index and the same
  distance metric when the server rejects hnsw with 422; older client
  versions without hfresh support re-raise the original error
- pin weaviate-client>=4.20.0, the first release that can express the
  hfresh index. The previous unpinned requirement silently resolved to
  4.16.2 (which always sends hnsw, even with no index config) because
  weaviate-client 4.17+ requires grpcio<1.80 while the node pinned
  grpcio<=1.81.1; align the grpc pins below 1.80

Verified end to end against a Weaviate Cloud hfresh-only cluster via
the downstream port of this fix: the collection is created with the
HFRESH index type and all objects are inserted with zero batch
failures.

* fix(store_weaviate): tolerate weaviate packages without the exceptions module

The unit tests load the node against the mock weaviate package, which
did not provide weaviate.exceptions, so the new import broke every
store_weaviate test and the dynamic node case. Guard the import with a
fallback exception class (never raised by a real client) and add the
exceptions module to the test mock so the 422 retry path stays
testable.

* fix(store_weaviate): narrow the hfresh fallback and keep the actionable error

The retry fired for any 422 and reused the configured distance metric, so a
metric hfresh cannot take produced a second, more confusing rejection that hid
the original "hnsw not allowed" message.

- Skip the retry when the metric is not one hfresh supports (cosine and
  l2-squared), and surface the hnsw error instead.
- When both attempts fail, report the hnsw error and attach the hfresh failure
  as its cause. A 422 that was never about the index type (a rejected property,
  say) now explains itself instead of being masked by the workaround.
- Lift the arguments shared by both create() calls into one dict so only
  vector_index_config varies.

Adds coverage for the fallback, which had none: the hnsw happy path, the 422
retry, non-422 errors passing straight through, unsupported metrics skipping
the retry, and the double-failure case reporting the original error. Also adds
hfresh to the weaviate config mock so the fallback is reachable in tests.

* fix(store_weaviate): unpin grpcio and drop the unreachable hfresh guard

The grpc ceiling was repo-wide, not weaviate-local: lock-node-deps compiles
every node's requirements into one solve, and store_milvus and store_qdrant
list grpcio bare and inherit it. Lowering the bound here moved it for all of
them. weaviate-client carries the ceiling it needs in its own metadata, so the
grpc packages go back to bare and constraints.lock resolves them once.

The hfresh guard was dead on both paths it was written for: requirements floors
the client at 4.20.0, which has hfresh, and the test mock defines it too. Same
for the ImportError fallback around UnexpectedStatusCodeError -- the mock ships
weaviate/exceptions.py. Both removed.

* fix(store_weaviate): keep the hnsw error when the hfresh retry fails on the network

The retry handler caught UnexpectedStatusCodeError only. WeaviateConnectionError
and WeaviateTimeoutError derive from WeaviateBaseError instead, so a connection
drop or timeout during the hfresh attempt propagated and replaced the 422 that
explains the real cause -- leaving the user a network message and no mention of
hnsw not being allowed.

Catches the base class, so the workaround's own failure stays context and the
actionable error is still the one reported.

The test mock mirrored only the one exception class and flattened the hierarchy;
it now carries WeaviateBaseError with the connection and timeout errors under
it, the way the real package has them, so a handler catching the base is
exercised rather than assumed.

---------

Co-authored-by: ksaurabhAparavi <ksaurabhAparavi@users.noreply.github.com>

* fix(store_chroma): connect to Chroma Cloud with TLS, token auth, and tenant/database (#1768)

* fix(store_chroma): connect to Chroma Cloud with TLS, token auth, and tenant/database

The connection branch was selected on the 'profile' key, which
Config.getNodeConfig consumes while merging the selected profile's
contents, so any service/autopipe configured run silently took the
local branch and sent plain unauthenticated HTTP to the TLS-only
Chroma Cloud endpoint. The server either answers with something the
HTTP parser rejects (surfaces as 'illegal request line') or never
answers, and because the chromadb client hardcodes timeout=None on its
session and issues its first request during construction, the whole
pipeline then hangs indefinitely.

- select the branch on 'mode' like the weaviate node, keeping
  'profile' as a fallback for direct configurations
- cloud branch connects with ssl, sends the API key in the
  x-chroma-token header, and passes the tenant and database that
  Chroma Cloud requires; both are new optional cloud profile fields
- give every httpx session created by chromadb's transport module a
  30s connect / 120s request timeout so an unresponsive connection
  surfaces as an error instead of hanging the pipeline

Verified end to end against Chroma Cloud (api.trychroma.com) with a
real tenant via the downstream port of this fix: 17 objects published,
collection contents confirmed through the API.

* docs(store_chroma): document the Cloud connection (TLS, x-chroma-token, tenant/database)

The README still described the removed Settings/TokenAuthClientProvider
flow. Document the TLS-only connection, the x-chroma-token API key
header, the tenant and database that ChromaDB Cloud requires, and the
HTTP session timeout.

* fix(store_chroma): keep the cloud host default empty

Prefilling api.trychroma.com in the connection form is wrong for
self-hosted TLS servers using the cloud profile; the host stays a
required user entry as before.

* fix(store_chroma): report when the HTTP timeout shim cannot be applied

The shim swallowed every failure, so if chromadb reorganised its transport
module the client silently went back to timeout=None and an unresponsive
server could hang the pipeline with nothing in the log to explain it.

Warn instead. The shim is hardening against an upstream default rather than a
feature, so a failure must not stop the node from connecting, but it does need
to be visible. Also spell out the patch's scope: it shadows `httpx` only inside
chromadb.api.fastapi, where this client builds its sessions, and preserves any
explicit timeout the caller passes. chromadb-client exposes no supported
timeout setting, which is why the default is installed this way.

* test(store_chroma): cover the cloud connection and the timeout shim

Every behaviour this branch adds was untested, and each of the three defects is
silent in its own way, so each gets a test that fails when the fix is undone.

Covers profile resolution (mode, the profile fallback, and mode winning over a
stale profile), the cloud transport (ssl, the x-chroma-token header, and no
header when there is no key), tenant and database (forwarded, omitted rather
than sent empty, one without the other), and the httpx shim -- a session built
with no timeout gets one, an explicit timeout survives, it applies once, other
httpx names still resolve through it, and a store still connects when the shim
cannot be installed at all.

Reverting the mode/profile fix fails 7 of them.

Also regenerates the params block, which was missing chroma.tenant and
chroma.database and still showed the pre-#2055 profile default.

* test(store_chroma): build the store inside the stub context

The shim installs itself while the Store is constructed, not while chroma.py is
imported, so the stub transport module has to still be in sys.modules at that
point. Scoping the stubs to the import alone left the shim tests passing only
when a previous test's stub happened to linger -- which held locally and failed
in CI, where chromadb is not installed.

The suite now passes in fixed and randomised order.

* fix(store_chroma): drop the timeout shim, make TLS configurable

Two changes from @asclearuc's review.

The shim is out. It monkey-patched chromadb's transport module, which has to be
re-verified on every client bump and turns into a silent no-op if the session
moves -- and it was never what made Chroma Cloud work. The hang it addressed
was the *consequence* of sending plain unauthenticated HTTP to a TLS-only
endpoint, so fixing the branch selection and the cloud connection removes it.
A server that is genuinely unresponsive still has no bound; that is a real gap
and belongs in its own change, with the finding that chromadb hardcodes
httpx.Client(timeout=None) in fastapi.py and exposes no supported setting.

TLS is now `chroma.ssl`, defaulting to true. Hardcoding it broke the other half
of this profile: a self-hosted server behind plain HTTP with token auth worked
through the removed Settings path and would otherwise have had no way back
except the local profile, which drops the API key. An unresolved '${...}'
placeholder keeps TLS on rather than silently downgrading to plaintext.

---------

Co-authored-by: ksaurabhAparavi <ksaurabhAparavi@users.noreply.github.com>

* fix(shell): don't paint the outage banner over a reachable server (#2199)

* fix(shell): don't paint the outage banner over a reachable server

A CORS-blocked or proxy-dropped bootstrap request and a genuinely dead
server both surface as the same opaque network TypeError, so an expired
stored token behind a CORS misconfig latched a kind:'network' failure
and painted "Can't reach the server" over the signed-out landing. That
turned a session expiry into two false staging-down reports in two days.

After latching a network failure in handleStoredTokenFailure, probe
/version same-origin (served directly on single-host deployments,
through the edge proxy on CDN-split ones). If the probe answers, the
server is reachable and the failure re-latches as session expiry with a
Sign in action; if it fails too, the network banner stands. The
downgrade is generation-guarded and only replaces a still-latched
network failure, so newer failures and successful reconnects are never
overwritten.

* fix(shell): key the outage-banner probe to the exact latched failure

Addresses review on #2183. The /version-probe downgrade guarded on
lastFailure.kind === 'network', which cannot tell one network failure
from another. A newer network failure latching in the same connection
generation while an older probe was still in flight would be overwritten
by that older probe's success and mislabeled a session expiry.

Capture the specific failure object the probe was launched for and
downgrade only while that exact object is still latched (identity, not
kind). updateConnectionStatus latches a fresh object per failure, so a
newer failure or a reconnect that cleared the latch swaps the reference
and is left untouched.

Adds a regression test for the same-generation stale-probe race.

* fix(shell): require the /version contract before downgrading a probe

Addresses the CodeRabbit review on this PR. initialize({ uri }) permits a
cross-origin serverUri, but the reachability probe always hits the page
origin's /version. A bare 200 from that origin does not prove the backend
is reachable, so a real cross-origin outage could be mislabeled a session
expiry and show the sign-in path instead of the network banner.

Parse the probe response and downgrade only when it is the server's own
/version contract (status 'OK' with a data.version string), via a new
isVersionResponse guard; a generic 200 from an unrelated origin no longer
qualifies. This keeps the supported topologies working (single-host is
same-origin; CDN-split edge-proxies /version to the backend) without a
same-origin(serverUri) gate, which would disable the downgrade on the
edge-proxied path.

Adds a regression test for the 200-but-not-/version case and updates the
existing probe tests to return the /version shape.

* feat(nodes): let db_hotdata agents in separate runs share one database (#2196)

* feat(nodes): let db_hotdata agents in separate runs share one database

Adds an optional database_id so a run attaches to an existing Hotdata
database instead of creating its own, and fixes the four defects that
only appear once agents in different runs write to the same database.
All four were reproduced against the live API, not inferred.

- attach: database_id set means attach and never delete; empty keeps
  today's behaviour exactly. get_schema falls back to information_schema
  over SQL, since a Database API Token gets 403 on GET /databases/{id}
  and so cannot read back default_connection_id.

- answers lane: agents told to call load_data on a shared database did
  so roughly one time in four, across two model families. The lane is
  the structural alternative, so it now extracts rows from what agents
  actually emit - fenced JSON, prose preamble, named row wrappers - and
  raises with the text quoted when there is genuinely no JSON.

- concurrency: Hotdata serialises writes per table and refuses a second
  writer with 409 RESOURCE_LOCKED. Eight simultaneous appends landed
  one. Those are now replayed, on any method except database creation,
  whose replay could orphan a billable database.

- shape: an append must carry every column the table has, or Hotdata
  refuses rather than dropping the column. A second producer therefore
  could not write at all. load_data now reads the table's live columns
  and projects onto them before retrying, for appends only - filling
  nulls on an upsert would wipe the columns it left out.

Also surfaces result_id on query results, deduplicated like the rows
path and withheld when the rows were truncated, so an agent cannot
materialise far more than it saw.

examples/symphony-test carries the acceptance evidence: 20 agents, 22
databases, five dependency-ordered waves, 8/8 checks in 84s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(nodes): address review findings on shared-database writes

- unwrap only the reserved `rows` envelope. `items`, `data` and `results`
  are plausible business column names, so unwrapping them turned one
  legitimate record with a nested list into several rows and dropped the
  column - a silent loss. Guessing wrong the other way is visible.

- re-read and re-project a refused append in a bounded loop. A concurrent
  producer can widen the table between our schema read and our retry, so
  the widened payload arrived stale and was refused for a different
  missing column, which one attempt surfaced as an unactionable failure.

- correct the database lifetime contract in the README. A database this
  node creates is deleted at that run's teardown, so its id cannot be
  handed to later runs; sharing requires a database created outside any
  pipeline. The load_data description no longer claims data lives only
  for the run, which is false in attach mode, and the ttl field is
  documented as not applying to an attached database.

Harness: verify beat identity before discarding it in the Composer
checks, surface duplicate publishes instead of collapsing them with
setdefault, bound the telemetry lock retry with a wall-clock deadline,
isolate room failures so one flaky agent cannot abort every check,
validate credentials by name rather than dying on a KeyError, and stop
indexing model-produced JSON blindly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(nodes): tighten shared-database docs and harness validation

- document that indexes on a shared database must be built outside the
  node. `build_index` needs a connection id an attached run cannot read,
  and there is no owning run to fall back to: a database created outside
  a pipeline has none, and one a run created dies with that run. The
  creator holds the id from the create response, so give it the REST
  call. Only index creation is closed to attached runs; bm25_search and
  vector_search work from all of them.

Harness:
- require every melody entry to yield a (beat, note) pair. Filtering the
  malformed ones out let a reply carrying the right 15 pairs plus a junk
  {"beat": 99} pass, because the junk was simply discarded.
- cap the telemetry lock sleep to the remaining budget. Checking the
  deadline only after a request meant a lock seen just before it slept an
  uncapped backoff past the budget and then issued another request.
- pick the credential to require from the model actually selected, so
  --gemini no longer fails on a missing OpenAI key.
- document ROCKETRIDE_ANTHROPIC_KEY as needed for --model anthropic.

- assert the widen retry count exactly. `<=` would have passed a
  regression that stopped retrying after the first attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(examples): stop beat validation crashing on a malformed beat

`str(beat).lstrip('-').isdigit()` strips every leading hyphen, so "--1"
passed the guard and then raised ValueError on int() - the guard admitted
exactly the input it existed to reject, and the run died instead of
failing the check. Replaced with a guarded conversion in one helper, so
every entry either yields a (beat, note) pair or the whole reply is
rejected. Covers "--1", floats, None, bools, lists and non-list payloads.

Also bound the telemetry lock retry by requests, not just by waiting:
capping the sleep to the remaining budget still let a sleep landing
exactly on the deadline fall through and issue one more load. Re-check
the deadline after sleeping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(examples): measure 20 agent nodes in one pipeline against 20 runs

Generates and runs the canvas-native shape - all twenty agents in one
64-node graph, each with its own database, all publishing into one
shared db_hotdata node through their answers lanes.

It works: the engine opens the graph, one node accepts the answers lane
from twenty different agents, and all twenty verdicts are correct.

It is also serial. 1/4/8/20 agents take 7s/28s/57s/152s - about 7.6s
each, near-perfectly linear - against ~83s for the same twenty as twenty
separate runs. Twenty agent nodes on one canvas is not twenty parallel
agents, which is worth knowing before building a demo on the version
that looks right. A per-agent LLM node changes nothing (55s vs 57s at
eight), so a contended LLM node is not the cause; the mechanism is not
established here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(examples): keep the pipeline templates, drop the harness

Ariel is right: examples/ holds .pipe templates and their resources, no
Python anywhere, and no workflow reads the folder. Seven scripts needing
live Hotdata credentials would sit there with no CI behind them and no
signal when they rot - and they had grown to 7,438 of this PR's 8,853
added lines.

Keeps the three .pipe templates, which are what examples/ is for, and
renames the folder to `symphony/` since nothing under it tests anything
any more. The scripts and the findings write-up move to a gist linked
from the PR description.

Nothing is lost from the repo: every durable finding these scripts
produced is already in the node's own README - answers-lane publishing,
one row shape per table, the RESOURCE_LOCKED retry, attached-database
lifetime and the external index flow.

Also removes the hydra/ and blast-radius/ sections from examples/README.
Those folders are not in this PR, so the README documented two
directories that do not exist in the repository.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(examples): drop the examples changes from this PR

Keeps the change to what it is about: the db_hotdata node. examples/ is
now byte-identical to develop.

The pipeline templates and the harness that produced these findings live
in the companion gist linked from the PR description, and every durable
finding is in the node's own README.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: mithileshgau <mithilesh.gaurihar@rocketride.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tool_slack): keep subtype and bot_id so channel_history records can be classified (#1827)

* fix(tool_slack): keep subtype and bot_id so channel_history records can be classified

conversations.history interleaves system events with real messages, and
subtype is the only field that separates them. _clean_message stripped it,
so a channel_join -- which carries a ts, a user, and a text of
"<@U0123ABCD> has joined the channel" -- arrived byte-for-byte in the shape
of a real message, with no way to filter it and no way to recover the
distinction. In one channel read, 30 records came back for 19 actual
messages: 37% of what an agent was handed as conversation history was
join/leave noise with plausible timestamps and user ids.

bot_id was dropped the same way. check_connection already returns the
caller's own bot_id, so there was a value to compare against and nothing to
compare it to -- and an agent posting into a channel it also reads sees its
own output as new input on the next run. Bot messages often carry no user
at all, so `user` cannot filter them either.

Both fields are already in the raw payload; they were being deleted on the
way out. Both are absent on ordinary user messages, so the shape is
unchanged for existing callers -- pinned by a test.

The allowlist approach is right; it just needs to include the fields that
classify a record, not only the ones that carry its content.

README updated: the returned-field list is the tool's public contract.

Refs #1826

* docs(tool_slack): document the new fields on channel_history itself

Per CodeRabbit, two things.

channel_history's own Returns block still described only ts/user/text and
thread_ts. _clean_message and the README were updated with the new fields but
the method docstring was not, which is the one a caller reads first.

The README also listed `bot_message` among the system-event subtype examples.
It is not a system event in the channel_join sense -- it classifies a
bot-authored post. Now described separately from the join/leave/topic events,
alongside the bot_id guidance it actually relates to.

Refs #1826

* fix(tool_slack): correct channel_history subtype guidance and surface it to the agent

The tool description still promised only ts, user and text, so the new
discriminators never reached the LLM. It now carries the contract.

The guidance was also wrong: treating any subtype as noise drops real
messages (file_share, thread_broadcast, me_message), and bot_message is
absent when an app posts under its own bot identity, so bot_id is the
reliable discriminator. Adds tests for both shapes.

---------

Co-authored-by: Dylan Savage <dylanmsavage@hotmail.com>

* docs: fix nonexistent 'rrext_store get_all_projects' CLI example (#1613)

* docs: fix nonexistent 'rrext_store get_all_projects' CLI example

The Python client README and client-python docs both showed
'rocketride rrext_store get_all_projects' in the CLI examples block,
but no such command exists: the CLI registers the command as 'store'
(rrext_store is the underlying wire command name), and its subcommands
are dir, type, write, rm, mkdir, and stat — there is no
'get_all_projects' subcommand.

Replace the line with 'rocketride store dir /', matching the store
examples documented in packages/docs/content-static/cli.mdx.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: list store sub-commands under the CLI examples (review follow-up)

Review on #1613 suggested a newcomer-facing note for the store example.
The suggested wording ('store dir / is an alias of store list /') is not
accurate — no 'store list' subcommand or alias exists — so instead add a
correct note under the CLI block in both files listing the six real
sub-commands (dir, type, write, rm, mkdir, stat) and pointing to
'rocketride store --help'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: nihalnihalani <24360630+nihalnihalani@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* feat(cli,ci): 'rocketride validate' subcommand (Python + TS) and validate-pipes GitHub Action (#1573)

* fix(ai): wrap rrext_validate payload in the {'pipeline': ...} envelope validatePipeline expects

The DAP rrext_validate handler passed the pipeline config flat to
rocketlib validatePipeline, but the C++ binding requires the config under
a top-level 'pipeline' key — the envelope the HTTP path (modules/pipe
pipe_Validate) already builds. As a result every rrext_validate call,
including SDK client.validate(), failed with ccode 40 "'pipeline' is
missing or invalid" regardless of the config's actual validity (verified
against the published engine image v3.3.1.35 and current source).

Wrap the resolved payload exactly like pipe_Validate does. Existing
handler tests updated to assert the envelope, plus a regression test
pinning the wire shape.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(cli): add 'rocketride validate' subcommand to the Python and TypeScript CLIs (#1572)

Surfaces the existing SDK validate() (rrext_validate) in both CLIs so
.pipe files can be checked from a terminal or CI without executing them:

  rocketride validate <files...> [--source <id>] [--json]

- Files: literal paths and/or glob patterns, expanded in-CLI (recursive,
  sorted, deduped) so behavior is identical on shells without globbing.
- Accepts either the flat pipeline config or the { "pipeline": {...} }
  wrapper (unwrapped exactly like client.use()); non-object JSON is
  rejected locally before any server call.
- --json emits a single machine-readable document (per-file errors and
  warnings passed through verbatim from the engine, plus a summary).
- Exit codes: 0 all valid; 1 at least one file invalid; 2 usage error,
  connection failure, or no file received a server verdict.
- Connection handling, argument style, and output formatting mirror the
  existing subcommands in each CLI; no new dependencies.

Both CLIs implement the identical contract (byte-aligned human output
and JSON shape); covered by 14 pytest and 27 jest unit tests with a
mocked client — no server required.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ci): add validate-pipes composite GitHub Action

Reusable composite action that gates .pipe files in CI: sets up Python,
installs the rocketride CLI (fails fast with a clear message when the
installed release…
AviNormie pushed a commit to AviNormie/rocketride-server that referenced this pull request Nov 9, 2026
…an be classified (rocketride-org#1827)

* fix(tool_slack): keep subtype and bot_id so channel_history records can be classified

conversations.history interleaves system events with real messages, and
subtype is the only field that separates them. _clean_message stripped it,
so a channel_join -- which carries a ts, a user, and a text of
"<@U0123ABCD> has joined the channel" -- arrived byte-for-byte in the shape
of a real message, with no way to filter it and no way to recover the
distinction. In one channel read, 30 records came back for 19 actual
messages: 37% of what an agent was handed as conversation history was
join/leave noise with plausible timestamps and user ids.

bot_id was dropped the same way. check_connection already returns the
caller's own bot_id, so there was a value to compare against and nothing to
compare it to -- and an agent posting into a channel it also reads sees its
own output as new input on the next run. Bot messages often carry no user
at all, so `user` cannot filter them either.

Both fields are already in the raw payload; they were being deleted on the
way out. Both are absent on ordinary user messages, so the shape is
unchanged for existing callers -- pinned by a test.

The allowlist approach is right; it just needs to include the fields that
classify a record, not only the ones that carry its content.

README updated: the returned-field list is the tool's public contract.

Refs rocketride-org#1826

* docs(tool_slack): document the new fields on channel_history itself

Per CodeRabbit, two things.

channel_history's own Returns block still described only ts/user/text and
thread_ts. _clean_message and the README were updated with the new fields but
the method docstring was not, which is the one a caller reads first.

The README also listed `bot_message` among the system-event subtype examples.
It is not a system event in the channel_join sense -- it classifies a
bot-authored post. Now described separately from the join/leave/topic events,
alongside the bot_id guidance it actually relates to.

Refs rocketride-org#1826

* fix(tool_slack): correct channel_history subtype guidance and surface it to the agent

The tool description still promised only ts, user and text, so the new
discriminators never reached the LLM. It now carries the contract.

The guidance was also wrong: treating any subtype as noise drops real
messages (file_share, thread_broadcast, me_message), and bot_message is
absent when an app posts under its own bot identity, so bot_id is the
reliable discriminator. Adds tests for both shapes.

---------

Co-authored-by: Dylan Savage <dylanmsavage@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation module:nodes Python pipeline nodes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(tool_slack): channel_history drops subtype and bot_id, so system events and the bot's own posts are indistinguishable from messages

3 participants