Skip to content

feat(engine): add OpenTelemetry tracing and Prometheus metrics - #498

Closed
nihalnihalani wants to merge 6 commits into
rocketride-org:developfrom
nihalnihalani:feature/observability-otel-prometheus
Closed

nihalnihalani wants to merge 6 commits into
rocketride-org:developfrom
nihalnihalani:feature/observability-otel-prometheus

Conversation

@nihalnihalani

@nihalnihalani nihalnihalani commented Mar 30, 2026 •

Copy link
Copy Markdown
Collaborator

#Hack-with-bay-2

Contribution Type

Feature — Add OpenTelemetry distributed tracing and Prometheus metrics export

Problem / Use Case

RocketRide Server has an internal MetricsManager (packages/ai/src/ai/web/metrics/metrics.py) that tracks per-task timers and counters, but this data is only accessible within the engine. There is no way to:

  • Export metrics to external monitoring systems (Prometheus, Grafana, Datadog)
  • Trace requests across pipeline stages (distributed tracing)
  • Monitor LLM API costs, latency, and error rates in real-time
  • Set up alerting on pipeline performance degradation

Proposed Solution

  1. Prometheus metrics (prometheus_metrics.py) — 9 metrics covering pipelines, LLM requests/tokens, vector DB ops, HTTP requests. Custom histogram buckets for LLM durations (0.1s to 120s).
  2. OpenTelemetry tracing (tracing.py) — Configurable OTLP/console/none exporter with thread-safe setup via threading.Lock. Auto-instruments FastAPI.
  3. Metrics middleware (middleware.py) — Automatic HTTP request instrumentation with URL path normalization to prevent label cardinality explosion (UUIDs and numeric IDs replaced with /{id}).
  4. /metrics endpoint (metrics_endpoint.py) — Prometheus text exposition format, configurable public/private access with security documentation.
  5. 27 unit tests covering all components.

Value Added

  • Production observability: Prometheus scraping, Grafana dashboards, PagerDuty alerts
  • Cost tracking: rocketride_llm_tokens_total by provider/model enables per-model cost analysis
  • Performance monitoring: Pipeline duration histograms identify slow pipelines
  • Zero-config start: OTEL_EXPORTER_TYPE=none (default) adds no overhead

Why This Feature Fits This Codebase

The existing MetricsManager at packages/ai/src/ai/web/metrics/metrics.py already tracks timers and counters per-task/per-pipe (lines 96-423). This PR extends that foundation with industry-standard export formats without modifying the existing class.

The FastAPI server at packages/ai/src/ai/web/server.py already uses middleware (AuthMiddleware at line 20). The new MetricsMiddleware follows the identical Starlette BaseHTTPMiddleware pattern. URL path normalization uses Starlette's route matching (route.matches(scope)) to resolve paths to templates, preventing the cardinality explosion that would occur with dynamic path parameters like /pipeline/{id}/status.

The /metrics endpoint follows the same registration pattern as existing endpoints in packages/ai/src/ai/web/endpoints/ (ping, version, status, shutdown).

Validation

$ pytest test/nodes/test_observability.py -v
===== 27 passed =====

Tests verify: counter increments, histogram observations, gauge operations, Prometheus text format output, all 3 exporter types, middleware request recording for 200/404/500 status codes.

How This Could Be Extended

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added a /metrics endpoint and a metrics ASGI app; server now registers metrics middleware and route (public access configurable).
    • Automatic per-request telemetry middleware for normalized endpoint metrics and latency tracking.
  • Metrics

    • Added counters, histograms, and gauges for pipelines, LLM requests/tokens/duration, vector DB ops, active tasks, and HTTP requests.
  • Observability

    • Integrated OpenTelemetry tracing with configurable exporters and app instrumentation.
  • Tests

    • Added end-to-end observability tests covering metrics, endpoint, tracing, and middleware.

- Add Prometheus metrics: pipeline executions, LLM requests/tokens, vectordb ops, HTTP requests
- Add OpenTelemetry tracing with configurable OTLP exporter
- Add metrics middleware for automatic HTTP request instrumentation
- Add /metrics endpoint for Prometheus scraping
- Add comprehensive tests for all observability components (27 tests)
- Environment-driven configuration (OTEL_EXPORTER_OTLP_ENDPOINT, ROCKETRIDE_METRICS_PUBLIC, etc.)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Mar 30, 2026 •

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds Prometheus metrics, a metrics exposition endpoint, request-observability middleware, and OpenTelemetry tracing bootstrap/shutdown; integrates middleware and metrics route into the server and includes end-to-end tests for metrics, middleware, and tracing.

Changes

Cohort / File(s) Summary
Prometheus metrics & helpers
packages/ai/src/ai/web/metrics/prometheus_metrics.py, packages/ai/src/ai/web/metrics/requirements.txt
New shared Prometheus REGISTRY and instruments (counters, histograms, gauge), helpers generate_metrics(), get_metrics_content_type(), get_metrics_app(), and requirements for prometheus_client and OpenTelemetry packages.
HTTP Middleware
packages/ai/src/ai/web/metrics/middleware.py
Added MetricsMiddleware (Starlette BaseHTTPMiddleware) that normalizes request paths, starts OpenTelemetry spans, times requests, records HTTP_REQUESTS and HTTP_REQUEST_DURATION with labels, sets span attributes, and records metrics for successful and exceptional responses.
Metrics endpoint
packages/ai/src/ai/web/endpoints/metrics_endpoint.py
New is_metrics_public() reads ROCKETRIDE_METRICS_PUBLIC and metrics_endpoint() returns Prometheus text bytes from generate_metrics() with media type from get_metrics_content_type(); both exported via __all__.
Tracing bootstrap
packages/ai/src/ai/web/metrics/tracing.py
New tracing utilities: setup_tracing(app=None) initializes a singleton TracerProvider and conditionally attaches exporters (otlp, console, none), optionally instruments FastAPI app, plus get_tracer() and shutdown_tracing().
Server integration
packages/ai/src/ai/web/server.py
Server now registers MetricsMiddleware, exposes GET /metrics gated by is_metrics_public(), and calls setup_tracing on startup and shutdown_tracing on shutdown.
Tests
test/nodes/test_observability.py
End-to-end tests for Prometheus metrics and exposition, /metrics ASGI response, tracing setup across exporter modes, tracer span usage and shutdown idempotency, and MetricsMiddleware.dispatch() behavior for success, exceptions (500), and non-200 responses.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Middleware as MetricsMiddleware
    participant App as FastAPI/AppHandler
    participant Prom as Prometheus\r\nRegistry
    participant OTel as OpenTelemetry\r\nTracer

    Client->>Middleware: HTTP request
    activate Middleware
    Middleware->>OTel: Start Span ("METHOD /normalized/path")\nattrs: http.method, http.url, http.route
    activate OTel
    Middleware->>App: call_next(request)
    activate App
    App-->>Middleware: Response (status) or Exception
    deactivate App
    Middleware->>Prom: Increment HTTP_REQUESTS (method, endpoint, status)
    Middleware->>Prom: Observe HTTP_REQUEST_DURATION (method, endpoint, duration)
    Middleware->>OTel: Set span attribute http.status_code
    OTel-->>Middleware: End Span
    deactivate OTel
    Middleware-->>Client: Return Response or propagate Exception
    deactivate Middleware
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐇 I hopped through routes and counted each beat,
Timed every hop, traced each fleet.
Prometheus hums, spans softly sing,
Metrics and traces — a springtime thing.
A rabbit cheers: observability takes wing!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 58.18% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'feat(engine): add OpenTelemetry tracing and Prometheus metrics' accurately and concisely summarizes the main change: adding observability via OpenTelemetry tracing and Prometheus metrics export.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/ai/src/ai/web/endpoints/metrics_endpoint.py`:
- Around line 31-35: The server bootstrap never registers the new metrics
endpoint, so /metrics is unreachable; in the startup code that calls
WebServer.add_route for endpoints (the block that currently adds '/ping',
'/use', '/shutdown', '/status', '/version'), add a call to
WebServer.add_route('/metrics', metrics_endpoint, ['GET'],
public=is_metrics_public()) and ensure metrics_endpoint and is_metrics_public()
are imported/available in that module so the route and its access toggle are
wired up at server initialization.
- Around line 47-53: The is_metrics_public() function currently treats an unset
ROCKETRIDE_METRICS_PUBLIC as true; change it so the default is private by making
the default value 'false' (or otherwise treat missing env var as false) and keep
accepting truthy strings like 'true','1','yes' to opt-in; update the function
body and its docstring accordingly and ensure the function name is unchanged so
route registration logic still calls is_metrics_public().

In `@packages/ai/src/ai/web/metrics/middleware.py`:
- Around line 29-33: WebServer.__init__ currently only registers AuthMiddleware
and CORSMiddleware so MetricsMiddleware never runs; update WebServer.__init__ to
call app.add_middleware(MetricsMiddleware) (registered after AuthMiddleware) so
the request path is resolved and HTTP metrics are recorded; locate the
middleware registration block where AuthMiddleware and CORSMiddleware are added
and append the MetricsMiddleware registration using the same app instance.
- Around line 51-52: Use the matched route pattern from request.scope['route']
instead of request.url.path for the endpoint label to avoid cardinality
explosion; in the middleware (around where method = request.method and path =
request.url.path and the other blocks at lines ~58-64 and ~74-77) retrieve route
= request.scope.get('route') and use route.path (or route.path_format) as the
endpoint template with a fallback to request.url.path when route is missing, and
replace all uses that set the endpoint/time-series label so parameterized routes
emit the template rather than concrete IDs.

In `@packages/ai/src/ai/web/metrics/requirements.txt`:
- Around line 1-5: The new observability deps listed in
packages/ai/src/ai/web/metrics/requirements.txt (prometheus_client,
opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-fastapi,
opentelemetry-exporter-otlp) are not declared in the package's primary install
manifest; add these exact package names (with appropriate version pins, e.g.,
>=0.20.0 / >=1.24.0 / >=0.45b0 as in the requirements file) to the package's
main packaging configuration (pyproject.toml [tool.poetry.dependencies] or
[project.dependencies], or setup.py install_requires) for the ai package so
imports succeed at runtime and CI installs include them.

In `@packages/ai/src/ai/web/metrics/tracing.py`:
- Around line 54-93: The tracing setup is never invoked because
setup_tracing(app) and shutdown_tracing() are not called from the FastAPI
lifespan handlers; modify the server lifecycle wiring to call
setup_tracing(self.app) during startup (inside the existing _on_startup /
startup callback flow) and call shutdown_tracing() during shutdown (inside
_on_shutdown / shutdown callback flow) so the TracerProvider in setup_tracing
and its FastAPI instrumentation are activated in production; look for
setup_tracing and shutdown_tracing and the methods _on_startup and _on_shutdown
in server.py to add these calls.
- Around line 65-85: The code currently calls
trace.set_tracer_provider(provider) unconditionally which cannot override an
existing global provider; change it to first inspect the current global via
trace.get_tracer_provider() and only call trace.set_tracer_provider(provider)
(and assign _tracer_provider = provider) if the current provider is the
default/no-op instance (or otherwise uninitialized); if a provider is already
set, skip replacing it and optionally log a debug message. Apply the same guard
around the other occurrence referenced (lines ~113-116) so both initialization
sites check trace.get_tracer_provider() before calling
trace.set_tracer_provider().

In `@test/nodes/test_observability.py`:
- Around line 279-287: The test patches tracing.FastAPIInstrumentor but
setup_tracing does a local import from opentelemetry.instrumentation.fastapi
inside the function, so update the test test_setup_tracing_with_fastapi_app to
patch the real symbol opentelemetry.instrumentation.fastapi.FastAPIInstrumentor
(e.g. with patch('opentelemetry.instrumentation.fastapi.FastAPIInstrumentor',
create=True)), set its instrument_app to a MagicMock, call
tracing.setup_tracing(app=mock_app), then assert the mocked instrument_app was
called and shutdown_tracing() as before; reference setup_tracing and
FastAPIInstrumentor to locate where to change the patch and add the assert.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f578f830-2921-49aa-9c47-d7b75f5d2075

📥 Commits

Reviewing files that changed from the base of the PR and between 4d473ca and 865b262.

📒 Files selected for processing (6)
  • packages/ai/src/ai/web/endpoints/metrics_endpoint.py
  • packages/ai/src/ai/web/metrics/middleware.py
  • packages/ai/src/ai/web/metrics/prometheus_metrics.py
  • packages/ai/src/ai/web/metrics/requirements.txt
  • packages/ai/src/ai/web/metrics/tracing.py
  • test/nodes/test_observability.py

Comment on lines +1 to +5
prometheus_client>=0.20.0
opentelemetry-api>=1.24.0
opentelemetry-sdk>=1.24.0
opentelemetry-instrumentation-fastapi>=0.45b0
opentelemetry-exporter-otlp>=1.24.0

@coderabbitai coderabbitai Bot Mar 30, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "## References to the new requirements file"
rg -n --hidden "packages/ai/src/ai/web/metrics/requirements\.txt|ai/web/metrics/requirements\.txt" . || true

echo
echo "## Manifest entries for the observability dependencies"
rg -n --hidden \
  -g 'pyproject.toml' \
  -g 'setup.py' \
  -g 'setup.cfg' \
  -g 'requirements*.txt' \
  -g 'Pipfile' \
  -g 'poetry.lock' \
  "prometheus_client|opentelemetry-api|opentelemetry-sdk|opentelemetry-exporter-otlp|opentelemetry-instrumentation-fastapi" . || true

Repository: rocketride-org/rocketride-server

Length of output: 594


Add dependencies to the main package manifest.

The new packages/ai/src/ai/web/metrics/requirements.txt is never referenced and the five observability dependencies (prometheus_client, opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-fastapi, opentelemetry-exporter-otlp) are not declared in any main packaging manifest (pyproject.toml, setup.py, or equivalent). These imports will fail at runtime unless the dependencies are added to the package's primary installation configuration.

🧰 Tools
🪛 OSV Scanner (2.3.5)

[HIGH] 1-1: requests 2.9.2: undefined

(PYSEC-2018-28)


[HIGH] 1-1: requests 2.9.2: undefined

(PYSEC-2023-74)


[HIGH] 1-1: requests 2.9.2: Requests vulnerable to .netrc credentials leak via malicious URLs

(GHSA-9hjg-9r4m-mvj7)


[HIGH] 1-1: requests 2.9.2: Requests Session object does not verify requests after making first request with verify=False

(GHSA-9wx4-h78v-vm56)


[HIGH] 1-1: requests 2.9.2: Requests has Insecure Temp File Reuse in its extract_zipped_paths() utility function

(GHSA-gc5v-m9x4-r6x2)


[HIGH] 1-1: requests 2.9.2: Unintended leak of Proxy-Authorization header in requests

(GHSA-j8r2-6x86-q33q)


[HIGH] 1-1: requests 2.9.2: Insufficiently Protected Credentials in Requests

(GHSA-x84v-xcm2-53pg)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/metrics/requirements.txt` around lines 1 - 5, The new
observability deps listed in packages/ai/src/ai/web/metrics/requirements.txt
(prometheus_client, opentelemetry-api, opentelemetry-sdk,
opentelemetry-instrumentation-fastapi, opentelemetry-exporter-otlp) are not
declared in the package's primary install manifest; add these exact package
names (with appropriate version pins, e.g., >=0.20.0 / >=1.24.0 / >=0.45b0 as in
the requirements file) to the package's main packaging configuration
(pyproject.toml [tool.poetry.dependencies] or [project.dependencies], or
setup.py install_requires) for the ai package so imports succeed at runtime and
CI installs include them.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai Good catch. The new requirements.txt in the metrics package is isolated and not referenced by the main package manifest (pyproject.toml / setup.py / requirements.txt). This means the observability dependencies (prometheus_client, opentelemetry-api, opentelemetry-sdk, etc.) will not be installed when the package is deployed. We will add these dependencies to the main package manifest to ensure they are installed as part of the standard deployment. Thank you for the investigation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate Limit Exceeded

@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 42 seconds before sending another message.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1479fa6 — added all five observability dependencies (prometheus_client, opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-fastapi, opentelemetry-exporter-otlp) to packages/ai/src/ai/web/requirements.txt, which is the canonical web package manifest used by the build system.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate Limit Exceeded

@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 12 seconds before sending another message.

Comment thread test/nodes/test_observability.py Resolved
- Use global prometheus_client.REGISTRY instead of custom CollectorRegistry
  so instrumentation libraries merge seamlessly and process/platform
  collectors are included automatically
- Normalize URL paths in middleware to route templates via Starlette
  route matching, with regex fallback for UUIDs and numeric segments,
  preventing Prometheus label cardinality explosion
- Add custom histogram buckets for LLM request durations (0.1s-120s)
  tuned for real-world LLM latency distribution
- Add threading.Lock around TracerProvider setup for thread safety
- Add rate limiting documentation for /metrics endpoint security

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (5)
packages/ai/src/ai/web/endpoints/metrics_endpoint.py (2)

31-35: ⚠️ Potential issue | 🔴 Critical

/metrics still needs server route registration.

This integration example is not wired in current packages/ai/src/ai/web/server.py route bootstrap, so the endpoint remains unreachable.

🔧 Suggested integration in packages/ai/src/ai/web/server.py
+from ai.web.endpoints.metrics_endpoint import metrics_endpoint, is_metrics_public
@@
         if register_standard_endpoints:
             self.add_route('/ping', ping, ['GET'], public=True)
             self.add_route('/use', use, ['POST'])
             self.add_route('/shutdown', shutdown, ['POST'])
+            self.add_route('/metrics', metrics_endpoint, ['GET'], public=is_metrics_public())
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/endpoints/metrics_endpoint.py` around lines 31 - 35,
The /metrics endpoint (metrics_endpoint) is not registered with the server
bootstrap so it is unreachable; update the server route registration code that
calls WebServer.add_route (or server.add_route) to add the '/metrics' GET route
pointing to metrics_endpoint and set the appropriate public/is_public flag;
locate the route-registration function (the server startup or route bootstrap
that registers other endpoints) and add a route registration for '/metrics'
using metrics_endpoint and the GET method so the endpoint becomes reachable.

50-64: ⚠️ Potential issue | 🟠 Major

Default /metrics visibility should be private.

Using 'true' as default makes internals publicly exposed unless explicitly hardened.

🔧 Suggested change
-    Controlled by ``ROCKETRIDE_METRICS_PUBLIC`` (default ``true``).
-    Set to ``false`` to require authentication for scraping.
+    Controlled by ``ROCKETRIDE_METRICS_PUBLIC`` (default ``false``).
+    Set to ``true`` to allow unauthenticated scraping.
@@
-    return os.environ.get('ROCKETRIDE_METRICS_PUBLIC', 'true').lower() in ('true', '1', 'yes')
+    return os.environ.get('ROCKETRIDE_METRICS_PUBLIC', 'false').lower() in ('true', '1', 'yes')
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/endpoints/metrics_endpoint.py` around lines 50 - 64,
The current default for ROCKETRIDE_METRICS_PUBLIC is 'true', which exposes
/metrics publicly; change the default to 'false' in the environment parse
expression (replace 'true' with 'false' in the return statement) and update the
surrounding docstring/comments to state that the default is private. Ensure any
related unit tests or configuration examples that assume a public default are
updated to expect a private default and verify that the function reading
ROCKETRIDE_METRICS_PUBLIC still accepts 'true','1','yes' as enabling public
metrics.
packages/ai/src/ai/web/metrics/middleware.py (1)

29-33: ⚠️ Potential issue | 🔴 Critical

Register MetricsMiddleware in WebServer.__init__.

packages/ai/src/ai/web/server.py currently shows only AuthMiddleware/CORSMiddleware; without adding this middleware, HTTP metrics in this file never execute.

🔧 Suggested integration in packages/ai/src/ai/web/server.py
+from ai.web.metrics.middleware import MetricsMiddleware
@@
         self.app.add_middleware(AuthMiddleware)
+        self.app.add_middleware(MetricsMiddleware)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/metrics/middleware.py` around lines 29 - 33, Add
MetricsMiddleware registration to WebServer.__init__ so HTTP metrics run; inside
the WebServer.__init__ method (where AuthMiddleware and CORSMiddleware are
added) call app.add_middleware(MetricsMiddleware) and ensure it is added after
AuthMiddleware so routes resolve correctly. Reference the MetricsMiddleware
class from ai.web.metrics.middleware and the app.add_middleware call in
WebServer.__init__; keep the existing AuthMiddleware and CORSMiddleware
registrations and insert the MetricsMiddleware addition immediately after the
auth middleware registration.
packages/ai/src/ai/web/metrics/tracing.py (2)

32-34: ⚠️ Potential issue | 🔴 Critical

Wire tracing into server lifespan; otherwise this module stays inactive.

setup_tracing()/shutdown_tracing() are documented here, but packages/ai/src/ai/web/server.py bootstrap/lifespan still doesn’t call them, so tracing won’t run in production.

🔧 Suggested integration in packages/ai/src/ai/web/server.py
+from ai.web.metrics.tracing import setup_tracing, shutdown_tracing
@@
 async def _on_startup(...):
+    setup_tracing(self.app)
     ...
@@
 async def _on_shutdown(...):
+    shutdown_tracing()
     ...
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/metrics/tracing.py` around lines 32 - 34, The tracing
module is never wired into the application lifecycle; call setup_tracing(app)
during application startup and shutdown_tracing() on application shutdown so the
instrumentation activates. Update the FastAPI/bootstrap lifespan in server.py to
invoke setup_tracing(app) when the app starts (e.g., in the lifespan context or
startup event) and ensure shutdown_tracing() is called on exit, passing the same
app/context so the tracing middleware and automatic FastAPI instrumentation are
registered and torn down properly.

93-95: ⚠️ Potential issue | 🔴 Critical

Don’t rely on re-calling set_tracer_provider after shutdown.

After shutdown_tracing() clears _tracer_provider, a later setup_tracing() creates a new provider and calls trace.set_tracer_provider(...) again. OpenTelemetry won’t replace an already-set global provider, which can leave local and global providers inconsistent.

In opentelemetry-python, what is the behavior of trace.set_tracer_provider() when called more than once? Does it replace the existing global provider?
🔧 Suggested guard
 from opentelemetry import trace
@@
-        trace.set_tracer_provider(provider)
-        _tracer_provider = provider
+        current = trace.get_tracer_provider()
+        if isinstance(current, TracerProvider):
+            _tracer_provider = current
+        else:
+            trace.set_tracer_provider(provider)
+            _tracer_provider = provider

Also applies to: 122-125

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/metrics/tracing.py` around lines 93 - 95, The code
calls trace.set_tracer_provider(provider) unconditionally in setup_tracing(),
which won't replace an already-set global provider after shutdown; update
setup_tracing() to first check trace.get_tracer_provider() and only call
trace.set_tracer_provider(provider) if the global provider is the default/noop
provider (or not already the same provider), and update shutdown_tracing() to
explicitly reset the global provider (e.g., call
trace.set_tracer_provider(trace.NoopTracerProvider()) or equivalent) and clear
_tracer_provider so subsequent setup_tracing() can safely install a new
provider; reference the functions setup_tracing(), shutdown_tracing(), the
module-level _tracer_provider, and
trace.get_tracer_provider()/trace.set_tracer_provider().
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/ai/src/ai/web/metrics/tracing.py`:
- Around line 72-75: The early return inside the _setup_lock block skips
FastAPIInstrumentor.instrument_app(app) when _tracer_provider already exists;
change the logic in the setup function(s) around the
_setup_lock/_tracer_provider checks so that if an app argument is provided you
still call FastAPIInstrumentor.instrument_app(app) even when _tracer_provider is
not None. Locate the branches that reference _setup_lock and _tracer_provider
(the first block around lines 72–75 and the similar block at 96–101) and modify
them to conditionally perform instrumentation when app is passed rather than
returning immediately, ensuring you only skip provider reinitialization but
always attempt to instrument the FastAPI app.

---

Duplicate comments:
In `@packages/ai/src/ai/web/endpoints/metrics_endpoint.py`:
- Around line 31-35: The /metrics endpoint (metrics_endpoint) is not registered
with the server bootstrap so it is unreachable; update the server route
registration code that calls WebServer.add_route (or server.add_route) to add
the '/metrics' GET route pointing to metrics_endpoint and set the appropriate
public/is_public flag; locate the route-registration function (the server
startup or route bootstrap that registers other endpoints) and add a route
registration for '/metrics' using metrics_endpoint and the GET method so the
endpoint becomes reachable.
- Around line 50-64: The current default for ROCKETRIDE_METRICS_PUBLIC is
'true', which exposes /metrics publicly; change the default to 'false' in the
environment parse expression (replace 'true' with 'false' in the return
statement) and update the surrounding docstring/comments to state that the
default is private. Ensure any related unit tests or configuration examples that
assume a public default are updated to expect a private default and verify that
the function reading ROCKETRIDE_METRICS_PUBLIC still accepts 'true','1','yes' as
enabling public metrics.

In `@packages/ai/src/ai/web/metrics/middleware.py`:
- Around line 29-33: Add MetricsMiddleware registration to WebServer.__init__ so
HTTP metrics run; inside the WebServer.__init__ method (where AuthMiddleware and
CORSMiddleware are added) call app.add_middleware(MetricsMiddleware) and ensure
it is added after AuthMiddleware so routes resolve correctly. Reference the
MetricsMiddleware class from ai.web.metrics.middleware and the
app.add_middleware call in WebServer.__init__; keep the existing AuthMiddleware
and CORSMiddleware registrations and insert the MetricsMiddleware addition
immediately after the auth middleware registration.

In `@packages/ai/src/ai/web/metrics/tracing.py`:
- Around line 32-34: The tracing module is never wired into the application
lifecycle; call setup_tracing(app) during application startup and
shutdown_tracing() on application shutdown so the instrumentation activates.
Update the FastAPI/bootstrap lifespan in server.py to invoke setup_tracing(app)
when the app starts (e.g., in the lifespan context or startup event) and ensure
shutdown_tracing() is called on exit, passing the same app/context so the
tracing middleware and automatic FastAPI instrumentation are registered and torn
down properly.
- Around line 93-95: The code calls trace.set_tracer_provider(provider)
unconditionally in setup_tracing(), which won't replace an already-set global
provider after shutdown; update setup_tracing() to first check
trace.get_tracer_provider() and only call trace.set_tracer_provider(provider) if
the global provider is the default/noop provider (or not already the same
provider), and update shutdown_tracing() to explicitly reset the global provider
(e.g., call trace.set_tracer_provider(trace.NoopTracerProvider()) or equivalent)
and clear _tracer_provider so subsequent setup_tracing() can safely install a
new provider; reference the functions setup_tracing(), shutdown_tracing(), the
module-level _tracer_provider, and
trace.get_tracer_provider()/trace.set_tracer_provider().
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2aa740ab-d8b6-4553-a4f1-56441e608b9a

📥 Commits

Reviewing files that changed from the base of the PR and between 865b262 and ac05893.

📒 Files selected for processing (4)
  • packages/ai/src/ai/web/endpoints/metrics_endpoint.py
  • packages/ai/src/ai/web/metrics/middleware.py
  • packages/ai/src/ai/web/metrics/prometheus_metrics.py
  • packages/ai/src/ai/web/metrics/tracing.py

nihalnihalani and others added 2 commits March 30, 2026 16:29
- Default ROCKETRIDE_METRICS_PUBLIC to 'false' (secure by default)
- Use request.scope.get('route') for endpoint label in metrics
  middleware instead of re-matching routes manually

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…heus integration

- Register /metrics endpoint in server bootstrap via add_route
- Register MetricsMiddleware in WebServer.__init__ after AuthMiddleware
- Wire setup_tracing(app) into _on_startup and shutdown_tracing() into _on_shutdown
- Fix tracing early return that skipped FastAPI app instrumentation when
  provider already existed; add idempotent _otel_instrumented guard
- Add observability deps to web/requirements.txt (canonical package manifest)
- Fix test_is_metrics_public_default to expect False (secure default)
- Fix test mock patch target for FastAPIInstrumentor to match actual import path
- Add scope/app attributes to middleware test mocks for _normalise_path compat

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@nihalnihalani

Copy link
Copy Markdown
Collaborator Author

Addressed CodeRabbit Review Comments (commit 1479fa6)

1. /metrics route not registered in server bootstrap (metrics_endpoint.py:35)
Fixed — added self.add_route('/metrics', metrics_endpoint, ['GET'], public=is_metrics_public()) in WebServer.__init__ alongside the other standard endpoints.

2. Default metrics to private (metrics_endpoint.py:53)
Already addressed in prior commit (0d1aa72). Test now correctly asserts is_metrics_public() returns False by default.

3. MetricsMiddleware not registered (middleware.py:33)
Fixed — added self.app.add_middleware(MetricsMiddleware) in WebServer.__init__ after AuthMiddleware.

4. Middleware test mocks missing scope/app (middleware.py:52)
Fixed — added scope and app attributes to mock request objects in all three middleware tests so _normalise_path resolves correctly.

5. Dependencies not in main package manifest (requirements.txt:5)
Fixed — added all five observability dependencies (prometheus_client, opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-fastapi, opentelemetry-exporter-otlp) to packages/ai/src/ai/web/requirements.txt (the canonical web package manifest).

6. Tracing not wired into FastAPI lifespan (tracing.py:93)
Fixed — setup_tracing(app=self.app) is now called in _on_startup() and shutdown_tracing() in _on_shutdown().

7. set_tracer_provider cannot safely replace global provider (tracing.py:85)
Fixed — restructured setup_tracing() so the TracerProvider is only created and set once (inside the _setup_lock block). The global trace.set_tracer_provider() is called exactly once.

8. Test patches wrong symbol for FastAPIInstrumentor (test_observability.py:287)
Fixed — changed patch target from tracing.FastAPIInstrumentor (which doesn't intercept the local import) to opentelemetry.instrumentation.fastapi.FastAPIInstrumentor.instrument_app and added assert_called_once_with verification.

9. App instrumentation skipped when provider already exists (tracing.py:75)
Fixed — moved FastAPI instrumentation outside the _setup_lock block so it runs even on subsequent calls. Added idempotent _otel_instrumented guard on app.state to prevent double-wrapping.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/ai/src/ai/web/metrics/tracing.py`:
- Around line 123-131: Protect accesses to _tracer_provider in shutdown_tracing
by acquiring the _setup_lock used by setup_tracing: inside the lock check if
_tracer_provider is not None, store it to a local variable and set the global
_tracer_provider to None, then release the lock and call
local_provider.shutdown(); this prevents races with setup_tracing while avoiding
holding the lock during the potentially slow shutdown call.

In `@packages/ai/src/ai/web/requirements.txt`:
- Around line 7-11: Update the OpenTelemetry entries in requirements.txt: bump
opentelemetry-instrumentation-fastapi to 0.61b0 and change opentelemetry-api and
opentelemetry-sdk from open-ended ">=..." to matching exact versions (e.g.,
opentelemetry-api==1.40.0 and opentelemetry-sdk==1.40.0) to ensure compatibility
and prevent version skew between the core API/SDK and the instrumentation
package.

In `@packages/ai/src/ai/web/server.py`:
- Around line 340-344: The call to shutdown_tracing() happens before awaiting
the user-provided shutdown callback (self._user_shutdown), which prevents any
spans emitted during that callback from being exported; modify the shutdown
sequence in the server shutdown routine so that you first await
self._user_shutdown() (if not None) and only then call shutdown_tracing(), i.e.,
move the shutdown_tracing() invocation to after the await of
self._user_shutdown() to allow user shutdown callbacks to use tracing; ensure
you preserve exception handling around the await so the tracer is still shut
down even if the user callback raises.

In `@test/nodes/test_observability.py`:
- Around line 79-142: Tests rely on the global prometheus_client REGISTRY
(prometheus_metrics.REGISTRY) which makes them order-dependent; update the tests
to use an isolated CollectorRegistry per test (or reset/unregister metrics
between tests) and register the metrics (PIPELINE_EXECUTIONS, LLM_REQUESTS,
LLM_TOKENS, VECTORDB_OPERATIONS, ACTIVE_TASKS, HTTP_REQUESTS) against that
registry so each test reads/writes only its own metrics; ensure each test
references REGISTRY.get_sample_value on the isolated registry instance (or
clears prometheus_metrics.REGISTRY) before incrementing to guarantee
deterministic, order-independent assertions.
- Around line 249-272: Tests leave global OpenTelemetry state set across runs;
ensure each test fully resets both the module-level _tracer_provider and the
global trace provider so setup_tracing() can reinitialize. Add an autouse pytest
fixture in test/nodes/test_observability.py that before each test sets
tracing._tracer_provider = None and calls trace.set_tracer_provider(None)
(import trace from opentelemetry.trace), yield, then call
tracing.shutdown_tracing() in teardown; this guarantees setup_tracing() and
shutdown_tracing() behave idempotently and isolates tests.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c9aefb2d-6bb7-47e3-96d8-4c8b300f4225

📥 Commits

Reviewing files that changed from the base of the PR and between ac05893 and 1479fa6.

📒 Files selected for processing (6)
  • packages/ai/src/ai/web/endpoints/metrics_endpoint.py
  • packages/ai/src/ai/web/metrics/middleware.py
  • packages/ai/src/ai/web/metrics/tracing.py
  • packages/ai/src/ai/web/requirements.txt
  • packages/ai/src/ai/web/server.py
  • test/nodes/test_observability.py

Comment thread packages/ai/src/ai/web/requirements.txt Outdated
Comment on lines +7 to +11
prometheus_client>=0.20.0
opentelemetry-api>=1.24.0
opentelemetry-sdk>=1.24.0
opentelemetry-instrumentation-fastapi>=0.45b0
opentelemetry-exporter-otlp>=1.24.0

@coderabbitai coderabbitai Bot Apr 3, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

What are the latest stable versions of opentelemetry-api, opentelemetry-sdk, and opentelemetry-instrumentation-fastapi Python packages, and what version compatibility is required between them?

💡 Result:

The latest stable versions are: - opentelemetry-api: 1.40.0 (released 2026-03-04) - opentelemetry-sdk: 1.40.0 (released 2026-03-04) - opentelemetry-instrumentation-fastapi: 0.61b0 (released 2026-03-04, beta status) opentelemetry-instrumentation-fastapi has beta status (PyPI status 4 - Beta) and no stable (1.x) release yet. Version compatibility: - opentelemetry-sdk 1.40.0 requires exactly opentelemetry-api ==1.40.0. - opentelemetry-instrumentation-fastapi 0.61b0 requires opentelemetry-api ~=1.12 (compatible with 1.40.0) and exact versions of related instrumentation packages like opentelemetry-instrumentation-asgi ==0.61b0. - Use matching major.minor versions from the core (opentelemetry-python) and contrib (opentelemetry-python-contrib) releases, which align monthly (e.g., core 1.40.0 with contrib 0.61b0). SDK pins exact API version; instrumentation uses loose ~= for API.

Citations:


Update OpenTelemetry instrumentation package version and ensure API/SDK version alignment.

The OpenTelemetry packages require careful version alignment. The instrumentation package (opentelemetry-instrumentation-fastapi>=0.45b0) is outdated—the latest version is 0.61b0 (released 2026-03-04), which aligns with the latest core packages (API/SDK 1.40.0). Additionally, opentelemetry-sdk requires an exact match with opentelemetry-api (e.g., SDK 1.40.0 requires API ==1.40.0), but your current constraints use >=, which could allow incompatible versions.

Update instrumentation to 0.61b0 and consider pinning API/SDK to the same version for stability (e.g., both 1.40.0), or use exact version constraints instead of >= to prevent version skew.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/requirements.txt` around lines 7 - 11, Update the
OpenTelemetry entries in requirements.txt: bump
opentelemetry-instrumentation-fastapi to 0.61b0 and change opentelemetry-api and
opentelemetry-sdk from open-ended ">=..." to matching exact versions (e.g.,
opentelemetry-api==1.40.0 and opentelemetry-sdk==1.40.0) to ensure compatibility
and prevent version skew between the core API/SDK and the instrumentation
package.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in afe6f7f. Added upper-bound version constraints to all OpenTelemetry dependencies (<2.0.0 for stable packages, <1.0.0 for the beta instrumentation package) to prevent breakage from future major version releases.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate Limit Exceeded

@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 26 seconds before sending another message.

Comment thread packages/ai/src/ai/web/server.py Resolved
Comment on lines +79 to +142
class TestPrometheusMetrics:
"""Validate that Prometheus metric objects are properly defined and usable."""

def test_pipeline_executions_counter_increments(self):
REGISTRY = prometheus_metrics.REGISTRY
before = REGISTRY.get_sample_value('rocketride_pipeline_executions_total', {'pipeline_name': 'test_pipe', 'status': 'success'}) or 0.0
prometheus_metrics.PIPELINE_EXECUTIONS.labels(pipeline_name='test_pipe', status='success').inc()
after = REGISTRY.get_sample_value('rocketride_pipeline_executions_total', {'pipeline_name': 'test_pipe', 'status': 'success'})
assert after == before + 1.0

def test_pipeline_executions_counter_error_label(self):
REGISTRY = prometheus_metrics.REGISTRY
before = REGISTRY.get_sample_value('rocketride_pipeline_executions_total', {'pipeline_name': 'err_pipe', 'status': 'error'}) or 0.0
prometheus_metrics.PIPELINE_EXECUTIONS.labels(pipeline_name='err_pipe', status='error').inc(3)
after = REGISTRY.get_sample_value('rocketride_pipeline_executions_total', {'pipeline_name': 'err_pipe', 'status': 'error'})
assert after == before + 3.0

def test_llm_requests_counter_increments(self):
REGISTRY = prometheus_metrics.REGISTRY
before = REGISTRY.get_sample_value('rocketride_llm_requests_total', {'provider': 'openai', 'model': 'gpt-4', 'status': 'success'}) or 0.0
prometheus_metrics.LLM_REQUESTS.labels(provider='openai', model='gpt-4', status='success').inc()
after = REGISTRY.get_sample_value('rocketride_llm_requests_total', {'provider': 'openai', 'model': 'gpt-4', 'status': 'success'})
assert after == before + 1.0

def test_llm_tokens_counter_by_type(self):
REGISTRY = prometheus_metrics.REGISTRY
before_in = REGISTRY.get_sample_value('rocketride_llm_tokens_total', {'provider': 'openai', 'model': 'gpt-4', 'token_type': 'input'}) or 0.0
before_out = REGISTRY.get_sample_value('rocketride_llm_tokens_total', {'provider': 'openai', 'model': 'gpt-4', 'token_type': 'output'}) or 0.0

prometheus_metrics.LLM_TOKENS.labels(provider='openai', model='gpt-4', token_type='input').inc(100)
prometheus_metrics.LLM_TOKENS.labels(provider='openai', model='gpt-4', token_type='output').inc(50)

after_in = REGISTRY.get_sample_value('rocketride_llm_tokens_total', {'provider': 'openai', 'model': 'gpt-4', 'token_type': 'input'})
after_out = REGISTRY.get_sample_value('rocketride_llm_tokens_total', {'provider': 'openai', 'model': 'gpt-4', 'token_type': 'output'})

assert after_in == before_in + 100.0
assert after_out == before_out + 50.0

def test_vectordb_operations_counter(self):
REGISTRY = prometheus_metrics.REGISTRY
before = REGISTRY.get_sample_value('rocketride_vectordb_operations_total', {'provider': 'milvus', 'operation': 'insert'}) or 0.0
prometheus_metrics.VECTORDB_OPERATIONS.labels(provider='milvus', operation='insert').inc()
after = REGISTRY.get_sample_value('rocketride_vectordb_operations_total', {'provider': 'milvus', 'operation': 'insert'})
assert after == before + 1.0

def test_active_tasks_gauge(self):
REGISTRY = prometheus_metrics.REGISTRY
prometheus_metrics.ACTIVE_TASKS.set(0)
prometheus_metrics.ACTIVE_TASKS.inc()
prometheus_metrics.ACTIVE_TASKS.inc()
val = REGISTRY.get_sample_value('rocketride_active_tasks')
assert val == 2.0

prometheus_metrics.ACTIVE_TASKS.dec()
val = REGISTRY.get_sample_value('rocketride_active_tasks')
assert val == 1.0

def test_http_requests_counter(self):
REGISTRY = prometheus_metrics.REGISTRY
before = REGISTRY.get_sample_value('rocketride_http_requests_total', {'method': 'GET', 'endpoint': '/ping', 'status_code': '200'}) or 0.0
prometheus_metrics.HTTP_REQUESTS.labels(method='GET', endpoint='/ping', status_code='200').inc()
after = REGISTRY.get_sample_value('rocketride_http_requests_total', {'method': 'GET', 'endpoint': '/ping', 'status_code': '200'})
assert after == before + 1.0

@coderabbitai coderabbitai Bot Apr 3, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Tests use shared Prometheus REGISTRY without isolation.

The prometheus_client REGISTRY is global and shared across tests. Since tests increment counters and don't reset them, tests depend on execution order and previous test state (using before/after pattern). This works but makes tests order-dependent. Consider using REGISTRY.get_sample_value(...) or 0.0 consistently (which you do) or creating isolated registries per test class for true isolation.

🧰 Tools
🪛 Ruff (0.15.7)

[error] 108-108: Possible hardcoded password assigned to argument: "token_type"

(S106)


[error] 109-109: Possible hardcoded password assigned to argument: "token_type"

(S106)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/nodes/test_observability.py` around lines 79 - 142, Tests rely on the
global prometheus_client REGISTRY (prometheus_metrics.REGISTRY) which makes them
order-dependent; update the tests to use an isolated CollectorRegistry per test
(or reset/unregister metrics between tests) and register the metrics
(PIPELINE_EXECUTIONS, LLM_REQUESTS, LLM_TOKENS, VECTORDB_OPERATIONS,
ACTIVE_TASKS, HTTP_REQUESTS) against that registry so each test reads/writes
only its own metrics; ensure each test references REGISTRY.get_sample_value on
the isolated registry instance (or clears prometheus_metrics.REGISTRY) before
incrementing to guarantee deterministic, order-independent assertions.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. The before/after pattern used throughout these tests already handles the shared REGISTRY correctly — each test measures deltas rather than absolute values, so execution order does not affect correctness. Creating isolated registries per test class would add complexity without meaningful benefit here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate Limit Exceeded

@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 23 seconds before sending another message.

Comment on lines +249 to +272
def test_setup_tracing_none_exporter(self):
"""With OTEL_EXPORTER_TYPE=none, setup should succeed without network calls."""
with patch.dict(os.environ, {'OTEL_EXPORTER_TYPE': 'none', 'OTEL_SERVICE_NAME': 'rocketride-test'}):
provider = tracing.setup_tracing()
assert provider is not None
tracing.shutdown_tracing()

def test_setup_tracing_console_exporter(self):
"""Console exporter should initialise without errors."""
with patch.dict(os.environ, {'OTEL_EXPORTER_TYPE': 'console', 'OTEL_SERVICE_NAME': 'rocketride-test'}):
provider = tracing.setup_tracing()
assert provider is not None
tracing.shutdown_tracing()

def test_get_tracer_returns_tracer(self):
"""get_tracer() should return a valid Tracer object."""
with patch.dict(os.environ, {'OTEL_EXPORTER_TYPE': 'none'}):
tracing.setup_tracing()
tracer = tracing.get_tracer('test-module')
assert tracer is not None
# Should be able to start a span
with tracer.start_as_current_span('test-span') as span:
assert span is not None
tracing.shutdown_tracing()

@coderabbitai coderabbitai Bot Apr 3, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Tests may not properly isolate tracing provider state.

Each test calls setup_tracing() but the module-level _tracer_provider persists across tests. Due to OpenTelemetry's singleton behavior for trace.set_tracer_provider(), once a provider is set globally, subsequent calls in other tests won't replace it. While shutdown_tracing() clears _tracer_provider, it doesn't reset the global trace provider.

Consider adding a fixture that fully resets tracing state between tests, or acknowledge that tests exercise the idempotent behavior rather than isolated initialization.

♻️ Suggested test fixture for isolation
`@pytest.fixture`(autouse=True)
def reset_tracing_state():
    """Reset tracing module state before each test."""
    tracing._tracer_provider = None
    yield
    tracing.shutdown_tracing()
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/nodes/test_observability.py` around lines 249 - 272, Tests leave global
OpenTelemetry state set across runs; ensure each test fully resets both the
module-level _tracer_provider and the global trace provider so setup_tracing()
can reinitialize. Add an autouse pytest fixture in
test/nodes/test_observability.py that before each test sets
tracing._tracer_provider = None and calls trace.set_tracer_provider(None)
(import trace from opentelemetry.trace), yield, then call
tracing.shutdown_tracing() in teardown; this guarantees setup_tracing() and
shutdown_tracing() behave idempotently and isolates tests.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in afe6f7f. Added an autouse fixture (reset_tracing_state) to TestOpenTelemetryTracing that resets _tracer_provider to None before each test and calls shutdown_tracing() in teardown. This ensures each test starts with a clean module-level state. Note: the global trace.set_tracer_provider() singleton behavior in OpenTelemetry is by design and cannot be fully reset without private API access, but clearing our module reference is sufficient for testing our initialization logic.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate Limit Exceeded

@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 18 seconds before sending another message.

nihalnihalani and others added 2 commits April 3, 2026 02:09
- Add lock protection to shutdown_tracing() to prevent race conditions
  with concurrent setup_tracing() calls
- Pin OpenTelemetry dependency upper bounds to prevent breaking changes
  from future major versions
- Reorder shutdown_tracing() after user callbacks in server shutdown so
  user-provided shutdown hooks can still emit spans
- Add autouse fixture to reset tracing state between tests for isolation
- Add test for concurrent shutdown safety

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Anchor _NUMERIC_RE and _UUID_RE to path segment boundaries to prevent
  partial matches like /pipeline/123abc/run -> /pipeline/{id}abc/run
- Add upper bounds to metrics requirements.txt to prevent breaking
  changes from major version bumps
- Fix middleware docstring to reflect LIFO middleware ordering (add
  before auth, not after)
- Add test for is_metrics_public(True) and path normalisation edge cases

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
packages/ai/src/ai/web/metrics/requirements.txt (1)

1-5: 🧹 Nitpick | 🔵 Trivial

This requirements file appears orphaned and may be dead code.

Based on the context snippets, packages/ai/src/ai/web/__init__.py only loads dependencies via depends(requirements) pointing to the parent requirements.txt. This nested metrics/requirements.txt is never referenced (no -r metrics/requirements.txt include, no depends() call).

Per the past review comments, the author confirmed adding these dependencies to packages/ai/src/ai/web/requirements.txt instead. Consider removing this orphaned file to avoid confusion about where dependencies are actually declared.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/metrics/requirements.txt` around lines 1 - 5, The
metrics/requirements.txt file appears orphaned: confirm whether the
prometheus/opentelemetry deps were intentionally moved to the parent
requirements.txt; if so, delete metrics/requirements.txt to avoid confusion and
run a repo-wide search for any references to "metrics/requirements.txt" to
ensure nothing breaks; if the file was meant to be included, instead add an
explicit include (e.g., -r metrics/requirements.txt) or a depends(...) call from
the parent requirements loader so the prometheus_client and opentelemetry
entries are actually consumed.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/ai/src/ai/web/metrics/middleware.py`:
- Around line 29-34: The docstring on MetricsMiddleware incorrectly instructs to
"add before the auth middleware" and misstates execution order; update the
docstring in MetricsMiddleware to reflect that Starlette middleware is LIFO and
that if AuthMiddleware is added first and MetricsMiddleware second (as done
where AuthMiddleware then MetricsMiddleware are registered), MetricsMiddleware
will run before AuthMiddleware, so route resolution may not yet be complete when
_normalise_path() runs and auth-failing requests will still be instrumented;
clarify the intended behavior (either instruct to add MetricsMiddleware before
AuthMiddleware to run after auth or explicitly document that MetricsMiddleware
runs before auth and handles missing route via request.scope['route'] fallback)
and reference MetricsMiddleware, AuthMiddleware and _normalise_path() in the
text.

---

Duplicate comments:
In `@packages/ai/src/ai/web/metrics/requirements.txt`:
- Around line 1-5: The metrics/requirements.txt file appears orphaned: confirm
whether the prometheus/opentelemetry deps were intentionally moved to the parent
requirements.txt; if so, delete metrics/requirements.txt to avoid confusion and
run a repo-wide search for any references to "metrics/requirements.txt" to
ensure nothing breaks; if the file was meant to be included, instead add an
explicit include (e.g., -r metrics/requirements.txt) or a depends(...) call from
the parent requirements loader so the prometheus_client and opentelemetry
entries are actually consumed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 33781512-7346-4e72-a19e-a1cc73c042e6

📥 Commits

Reviewing files that changed from the base of the PR and between afe6f7f and 0010f51.

📒 Files selected for processing (3)
  • packages/ai/src/ai/web/metrics/middleware.py
  • packages/ai/src/ai/web/metrics/requirements.txt
  • test/nodes/test_observability.py

Comment on lines +29 to +34
Usage — add *before* the auth middleware (Starlette middleware is LIFO,
so this will execute after auth has resolved the route path)::

from ai.web.metrics.middleware import MetricsMiddleware

app.add_middleware(MetricsMiddleware)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Docstring contradicts actual middleware registration order.

The docstring says "add before the auth middleware (Starlette middleware is LIFO, so this will execute after auth has resolved the route path)". However, in packages/ai/src/ai/web/server.py lines 135-136, AuthMiddleware is added first, then MetricsMiddleware is added second. With LIFO ordering, this means MetricsMiddleware executes before AuthMiddleware, not after.

This means:

  1. Route resolution may not have completed when _normalise_path() runs (the request.scope['route'] fallback handles this)
  2. Requests that fail auth (401/403) are still instrumented, which is actually desirable for observability

Consider updating the docstring to match reality or clarifying the intended behavior.

📝 Suggested docstring fix
-Usage — add *before* the auth middleware (Starlette middleware is LIFO,
-so this will execute after auth has resolved the route path)::
+Usage — add *after* AuthMiddleware in your middleware registration
+(Starlette middleware is LIFO, so MetricsMiddleware will execute first
+and capture all requests including auth failures)::
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/ai/src/ai/web/metrics/middleware.py` around lines 29 - 34, The
docstring on MetricsMiddleware incorrectly instructs to "add before the auth
middleware" and misstates execution order; update the docstring in
MetricsMiddleware to reflect that Starlette middleware is LIFO and that if
AuthMiddleware is added first and MetricsMiddleware second (as done where
AuthMiddleware then MetricsMiddleware are registered), MetricsMiddleware will
run before AuthMiddleware, so route resolution may not yet be complete when
_normalise_path() runs and auth-failing requests will still be instrumented;
clarify the intended behavior (either instruct to add MetricsMiddleware before
AuthMiddleware to run after auth or explicitly document that MetricsMiddleware
runs before auth and handles missing route via request.scope['route'] fallback)
and reference MetricsMiddleware, AuthMiddleware and _normalise_path() in the
text.

@stepmikhaylov stepmikhaylov added hackathon Hackathon project or submission zz: needs-refinement Deprecated 2026-08-05, do not use; refinement asks belong in comments zz: good-concept Deprecated 2026-08-05, do not use; reviewer sentiment belongs in comments labels Apr 6, 2026
@stepmikhaylov

Copy link
Copy Markdown
Collaborator

Thank you for the contribution and the detailed write-up.

After review, this PR is not ready for merge in its current state and is being closed to keep the PR list clean. The concept and proposed changes have been archived in #616 for further elaboration and future consideration.

nihalnihalani added a commit to nihalnihalani/rocketride-server that referenced this pull request Sep 4, 2026
…mented ingester protocol (rocketride-org#1609)

Maps the engine's monitor event stream to standard OTel telemetry with
zero engine/server changes — the client-side refinement invited by
tracking issue rocketride-org#616 after the engine-invasive rocketride-org#498 was archived.

- mapper.py: stateful FlowSpanMapper reconstructs span trees from
  apaevt_flow — task root span, pipe span per pipe id, component child
  spans paired by component identity (per the observability doc's
  warning against stack-position pairing); trace.error -> span status
  ERROR + exception event; apaevt_sse -> span events; unclosed spans at
  end/shutdown flagged rocketride.span.unclosed=true; per-pipe state is
  bounded. MetricsMapper maps apaevt_status_update snapshots to OTel
  instruments (object counts, rates, cpu/gpu memory + peaks).
- Semantic conventions per the July 2026 semantic-conventions-genai
  snapshot: gen_ai.provider.name (not the deprecated gen_ai.system),
  gen_ai.operation.name from the well-known list only, span names
  '{operation} {model}' style, INTERNAL span kind for in-process
  component executions.
- Privacy by default: lane payloads / prompt / result content never
  reach attributes or events unless include_content is set (8KB
  truncation cap when it is); enforced by a sentinel-scan test across
  all exported attributes and events.
- setup.py: TracerProvider/MeterProvider + OTLP http/protobuf exporters;
  ALL opentelemetry imports are lazy so the base package imports without
  the extra.

53 unit tests (InMemorySpanExporter/InMemoryMetricReader; importorskip
so the base CI matrix stays green), including a full replay of the
captured real-engine event fixture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
asclearuc pushed a commit that referenced this pull request Sep 15, 2026
…mented ingester protocol (zero engine changes) (#1612)

* feat(sdk): rocketride.otelbridge — OpenTelemetry export over the documented ingester protocol (#1609)

Maps the engine's monitor event stream to standard OTel telemetry with
zero engine/server changes — the client-side refinement invited by
tracking issue #616 after the engine-invasive #498 was archived.

- mapper.py: stateful FlowSpanMapper reconstructs span trees from
  apaevt_flow — task root span, pipe span per pipe id, component child
  spans paired by component identity (per the observability doc's
  warning against stack-position pairing); trace.error -> span status
  ERROR + exception event; apaevt_sse -> span events; unclosed spans at
  end/shutdown flagged rocketride.span.unclosed=true; per-pipe state is
  bounded. MetricsMapper maps apaevt_status_update snapshots to OTel
  instruments (object counts, rates, cpu/gpu memory + peaks).
- Semantic conventions per the July 2026 semantic-conventions-genai
  snapshot: gen_ai.provider.name (not the deprecated gen_ai.system),
  gen_ai.operation.name from the well-known list only, span names
  '{operation} {model}' style, INTERNAL span kind for in-process
  component executions.
- Privacy by default: lane payloads / prompt / result content never
  reach attributes or events unless include_content is set (8KB
  truncation cap when it is); enforced by a sentinel-scan test across
  all exported attributes and events.
- setup.py: TracerProvider/MeterProvider + OTLP http/protobuf exporters;
  ALL opentelemetry imports are lazy so the base package imports without
  the extra.

53 unit tests (InMemorySpanExporter/InMemoryMetricReader; importorskip
so the base CI matrix stays green), including a full replay of the
captured real-engine event fixture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(cli): add 'rocketride otel' subcommand + optional [otel] extra (#1609)

rocketride otel [--endpoint URL] [--protocol http|grpc] [--service-name]
[--headers k=v,...] [--include-content] [--no-metrics] + the standard
connection args. Subscribes via rrext_monitor with token '*' (the
observability doc's recommended ingestion scope, types TASK|SUMMARY|
FLOW|SSE), dispatches to the mappers, reconnects with capped backoff
and resubscribes (subscriptions are per-connection and not durable),
and flushes exporters on SIGINT/SIGTERM.

Endpoint resolution honors the standard OTLP env contract: explicit
--endpoint > OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (verbatim) >
OTEL_EXPORTER_OTLP_ENDPOINT (base URL + /v1/traces|/v1/metrics) >
default. Packaging: opentelemetry-sdk + OTLP http exporter live behind
an optional extra — the base install gains zero dependencies, and
running the subcommand without the extra exits 2 with the exact
"pip install 'rocketride[otel]'" hint before any connection attempt.

25 tests: config precedence matrix, dispatch/reconnect/resubscribe with
a fake client, shutdown flush ordering, missing-extra exit path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: OpenTelemetry bridge guide with Jaeger, Langfuse, and Datadog recipes

Setup guide covering the span model (task -> pipe -> component tree),
the privacy default and --include-content, the FLOW-requires-trace-level
callout, reconnect semantics, and copy-paste recipes with exact OTLP
endpoints/auth for Jaeger all-in-one, Langfuse (/api/public/otel, Basic
auth), and Datadog. Notes that Jaeger does not ingest OTLP metrics —
pair it with --no-metrics or point metrics at a collector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sdk): handle partial OpenTelemetry installs — guard the HTTP exporter import

CI exposed a real environment our matrix missed: opentelemetry-api/sdk
present as another package's transitive dependency (tool_daytona pulls
them in) while the OTLP HTTP exporter is absent. Two fixes:

- setup.py: the http trace/metric exporter imports are now guarded like
  the grpc ones — a partial install raises OtelNotInstalledError with
  the pip install 'rocketride[otel]' hint (CLI exit 2) instead of a raw
  ModuleNotFoundError. Verified live in a reproduced partial venv.
- test_otel_setup.py: importorskip targets the most specific module the
  tests exercise (opentelemetry.exporter.otlp.proto.http), so the module
  skips cleanly in CI's partial environment; plus a regression test that
  simulates the partial install via a None sys.modules entry and asserts
  the friendly error.

Partial venv: 63 passed + 1 module skip; full extra: 79 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sdk): OTel bridge review round — privacy-gate errors, run-id promotion, header precedence, shutdown ordering

CodeRabbit review round on #1612, each finding verified before acting:

- mapper: trace.error no longer bypasses the privacy gate — by default
  only the error signal exports (status ERROR, bare exception event,
  generic description); --include-content restores verbatim text, now
  subject to the same 8KB truncation strings previously skipped. The
  sentinel privacy test plants the sentinel in trace.error.
- mapper: a run first seen via flow events (before the task 'running'
  snapshot) is now promoted from its project|source fallback key to the
  canonical run id when it arrives — previously a second run state was
  created (broken parentage) and the orphaned fallback state leaked
  permanently. Confirmed with a failing repro; guard test ensures two
  distinct canonical runs sharing (project, source) never merge.
- config/setup: only --headers become explicit exporter headers; env
  headers are left to the SDK so OTEL_EXPORTER_OTLP_TRACES_HEADERS /
  _METRICS_HEADERS signal precedence works (it was silently defeated).
  Precedence matrix tests added.
- setup: meter provider shutdown wrapped in finally so metrics flush
  even when tracer shutdown raises.
- CLI: --trace-level none message now says flow tracing stays disabled;
  docs recipes pass collector auth via OTEL_EXPORTER_OTLP_HEADERS env
  instead of argv (shell-history hygiene); error-mapping doc wording
  matches the implementation (component leave events).

Full extra: 90 passed. Partial install (api/sdk without exporter):
69 passed + clean skip. ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sdk): absorb stranded fallback run state when canonical state precedes it

Follow-on to the run-promotion fix: an SSE event creates the canonical
run state WITHOUT registering the (project, source) alias (SSE bodies
carry no project/source per the protocol), so an id-less flow for the
same run could still open a parallel fallback-keyed state — and because
the canonical state already existed, the promotion branch was skipped,
stranding the fallback state (kept alive forever by snapshot
reconciliation) with duplicate spans.

_get_run now reconciles whenever an event carries both the canonical id
and project/source: no canonical state → promote (unchanged); canonical
state exists → _absorb_fallback_run migrates non-colliding pipes (open
spans re-stamped with the canonical run id), closes colliding pipes as
rocketride.span.unclosed=true, and drops the fallback key. Only
fallback-format keys are touched, so distinct canonical runs never
merge (guard test unchanged and green).

Both new regression tests failed before the fix. Full extra: 92 passed;
partial install: 71 passed + expected skip; ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sdk): stamp OTel spans with the engine's body.eventTime

The bridge was written against a monitor protocol that carried no event
timestamps, so every span used the bridge's own clock at event arrival and
the guide documented that as a limitation. Since #1661 the engine stamps
`body.eventTime` (epoch seconds, set once at ingress by the run-log
continuum) on every forwarded event, and derived flow events inherit their
source event's stamp — so the limitation is no longer true and the
documentation shipped with it was wrong against develop.

Span start/end and span-event timestamps now come from `body.eventTime`,
making durations engine-measured instead of bridge-observed: WebSocket
transport latency and event-loop scheduling no longer inflate every
component span.

The fallback is exactly the previous behaviour. A body with no usable
`eventTime` (an older engine, or a non-float/NaN/non-positive value) and
every span closed outside an event — shutdown, seeded-snapshot
reconciliation, tracked-run cap eviction — use `time.time_ns()`, the same
clock the SDK would have used. Because a run can therefore mix both
sources, each open span records its own start and a close is clamped to
never precede it, so no span can report a negative duration.

Metric points are unchanged: OpenTelemetry metric instruments take no
explicit timestamp, so they remain arrival-timed. That is now stated
rather than left implicit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(client-python): publish the OTel bridge guide and mirror its CLI entry

Two co-location gaps (AGENTS.md:5-28) and one stale claim.

The 369-line guide lived at docs/README-otel-bridge.md, and root docs/ is
not in the docs-site glob (packages/docs/scripts/lib/gather.js:25), so the
guide was GitHub-only. Moved to packages/client-python/docs/otel-bridge.md,
the tree already mounted at develop/python (packages/client-python/scripts/
tasks.js:236) alongside analytics.md and log.md, so it is gathered to the
site at /develop/python/otel-bridge. Its in-repo links become site routes
to match the sibling pages; the three references to it were repointed.

packages/client-python/docs/index.md is the site's Python page and mirrors
the PyPI README's ## CLI block line for line, but had no `rocketride otel`
row and none of the OpenTelemetry paragraph the README gained. Added both.

The guide's ## Limitations still said span timestamps are arrival time
because the wire carries none; the previous commit made spans engine-timed,
so that bullet now describes what actually remains bridge-timed (spans
closed outside an event, pre-continuum engines, and all metric points).

The Python blocks in the guide are also reformatted: ruff formats Markdown
code blocks as of 0.16, and CI's ruff-action is unpinned, so the file failed
`ruff format --check` today at either path. Repo-wide check is clean again.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(changelog): record the rocketride otel bridge under [Unreleased]

CHANGELOG.md [Unreleased] ### Added is the convention for user-visible
additions and is kept alphabetical by scope; a new CLI command and a new
optional extra qualify.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(client-python): name Grafana Tempo in the OTel backend recipes

Issue #1609's acceptance list says "Jaeger/Tempo", and the guide covered
Tempo only implicitly as "Grafana stacks". Tempo is the OTLP receiver a
Grafana user actually points the bridge at, and it needs one non-obvious
step (enable distributor.receivers.otlp) plus a separate metrics backend,
so it is worth naming rather than leaving a reader to infer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sdk): address CodeRabbit review round on the rebased head

Nine threads on the rebased head, all in the OTel bridge. Dispositions:

- docs/otel-bridge.md:151 (Sensitive Data Exposure, CWE-522) — FIXED. The
  Langfuse and LangSmith recipes now prompt for the keys (printf + read -rs,
  spelled identically in bash and zsh) instead of embedding placeholders that
  operators replace with live secrets in a command line, and point at a secret
  manager as the better source. The base64 pipeline is now
  `printf '%s' ... | base64 | tr -d '\r\n'`: real Langfuse keys make the
  joined string 85 characters, past GNU base64's 76-character wrap, and
  command substitution strips only trailing newlines, so the embedded break
  would have produced an invalid Authorization header.

- cli/commands/otel.py:207 (Sensitive Data Exposure, CWE-532) — FIXED. The
  startup line ran `config.endpoint` straight to stdout; an endpoint can carry
  userinfo credentials or a signed query string. It is now always passed
  through `redact_endpoint()`, which drops userinfo, query and fragment while
  keeping scheme/host/port and the ingest path operators actually verify.

- cli/main.py:466, otelbridge/config.py:177, otelbridge/setup.py:123
  (Cleartext Transmission, CWE-319) — FIXED, once, in one place.
  `validate_transport_security()` runs before any exporter exists (in
  `build_providers`, and again in the CLI so the failure is exit code 2 rather
  than a traceback): a credential-looking header name from `--headers` or any
  of the three OTEL_EXPORTER_OTLP_*HEADERS variables, plus a non-loopback
  cleartext endpoint for either signal, aborts startup. Header VALUES are
  never read or echoed — only names reach the message. Loopback stays exempt
  for local collectors; `--insecure` / ROCKETRIDE_OTEL_ALLOW_INSECURE=1 is the
  explicit opt-out and warns on stderr.

- otelbridge/config.py:189 (CWE-201, redirect header forwarding) — FIXED. The
  OTLP/HTTP exporters now get a session that never follows redirects.
  `requests` strips Authorization only when the redirect changes host, so a
  3xx replayed x-api-key to the target and even kept Authorization on an
  https -> http downgrade to the same host.

- otelbridge/config.py:177 (endpoint precedence) — FIXED. `from_args_env` no
  longer pre-reads OTEL_EXPORTER_OTLP_ENDPOINT into an explicit exporter
  endpoint: doing so overrode the signal-specific _TRACES_ENDPOINT /
  _METRICS_ENDPOINT, inverting the OTLP spec order — the same defect already
  fixed for the header variables in c23b312. The SDK now resolves all three,
  and `effective_endpoint()` mirrors its order read-only for the startup line
  and the security check.

- otelbridge/bridge.py:207 (signal ownership) — FIXED. `loop.add_signal_handler`
  replaces rather than chains, and asyncio exposes no way to restore the
  displaced callback, so the bridge no longer touches a signal the running
  loop already handles; `install_signal_handlers=False` turns registration off
  entirely for embedders.

- otelbridge/mapper.py:611 (unbounded run.pipes) — FIXED. MAX_TRACKED_RUNS did
  not bound pipes: a run that keeps receiving events is re-inserted as
  most-recently-eventful and never evicted, while every begin (or event for a
  never-begun pipe) adds a _PipeState that only a matching end removes, and a
  reconnect does not replay missed ends. MAX_TRACKED_PIPES_PER_RUN now closes
  the least-recently-eventful pipe as unclosed, with recency updated on pipe
  activity rather than inferred from insertion order.

- otelbridge/setup.py:202 (tracer leak) — FIXED. A failure anywhere in the
  metric half left the BatchSpanProcessor worker running with no handle to
  stop it, because `shutdown` had not been created yet. That path now shuts
  the tracer provider down before re-raising.

Tests: 32 added (131 in tests/test_otel_*.py, was 99), covering stdout
redaction, the refusal and its opt-out, env-var and signal-specific endpoint
coverage, the non-redirecting session, the tracer teardown, signal ownership
and the pipe cap's LRU behaviour.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(sdk): skip the loop signal-handler ownership test on Windows

asyncio's ProactorEventLoop raises NotImplementedError from
add_signal_handler, so the test cannot set up its precondition there.
The bridge itself already tolerates that platform (the other signal
tests pass on Windows); only the test needs the platform guard.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(cli): document the otel --insecure flag in the site CLI reference

The `otel` command grew an `--insecure` opt-out for the cleartext-credential
guard, but the docs-site "Key flags" table was never updated to match. Users
hitting exit code 2 from `rocketride otel --endpoint http://tempo:4318
--headers Authorization=...` had no way to learn why from the CLI reference —
the flag was only documented in the client-python OTel bridge guide.

Add the missing row, naming the `ROCKETRIDE_OTEL_ALLOW_INSECURE` env var
alongside the flag so the site table matches otel-bridge.md and the argparse
help text.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sdk): declare the otel test dependency and stop orphaning bridge providers

Two review findings on the `rocketride otel` bridge.

1. The opentelemetry dependency of the bridge tests was inherited, not
   declared. tests/test_otel_mapper.py and tests/test_otel_setup.py opened
   with pytest.importorskip('opentelemetry'), and nothing in the test path
   installs the 'otel' extra — the packages reached the engine interpreter
   only as a transitive dependency of the crewai node. Dropping or repinning
   that node would have turned ~1450 lines of mapper/setup coverage into a
   silent skip with CI still green.

   CI never pip-installs the package (pytest runs on the engine interpreter
   importing the package from the source checkout via tests/conftest.py), so the extras alone cannot fix this. The
   deps are added to packages/server/build-requirements.txt, which
   server:setup-test-deps installs through depends() under the merged
   constraints — the one install step that path goes through — and the
   'dev'/'test' extras now pull 'rocketride[otel]' so a source checkout
   matches. Both importorskip guards become hard imports: a missing
   dependency now fails collection instead of skipping.

2. build_providers() always built both provider halves and adopted its
   shutdown only when shutdown_fn was None, so an embedder passing its own
   shutdown_fn left a TracerProvider and its BatchSpanProcessor worker
   thread with nothing to stop them — and the half the caller had already
   supplied via a factory was built, wired and discarded.

   build_providers() takes with_tracer / with_meter (defaulting to the old
   behaviour) and run_bridge asks only for the half it is missing; a
   caller-supplied shutdown_fn is now chained in front of the providers'
   shutdown — caller first, providers in a finally — instead of replacing
   it. The CLI path passes neither argument and is unaffected.

Regression tests cover chain order, the provider shutdown surviving a
raising caller, and which half is built for each factory combination;
test_otel_bridge.py stays runnable without the extra. The embedded-use
ownership rules are documented in the bridge doc.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* build(client-python): install the otel test deps in the test step, not the engine

Review follow-up: the previous commit declared opentelemetry-sdk and
opentelemetry-exporter-otlp-proto-http in packages/server/build-requirements.txt.
That file is installed by server:setup-test-deps on every engine build, so every
engine install carried the OTel SDK and the OTLP exporter whether or not anyone
ran the bridge — which made the summary's "zero new base dependencies" true of
the package but not of the engine.

Move the cost to the step that needs it. A new client-python:setup-test-deps
action installs packages/client-python/tests/requirements.txt into the engine
interpreter through depends(), and runs only inside client-python:test, between
the parallel build group and the test-server bracket:

  server:build -> [nodes:build | ai:build | client-python:build]
               -> client-python:setup-test-deps
               -> py-test-server:setup -> client-python:run-pytest -> teardown

That ordering puts it after the node/ai requirement files land in the dist, so
depends() resolves against the complete constraint set, and before the bracket,
so a cold install does not hold the test server open. This mirrors the existing
packages/ai/tests/requirements.txt pattern (packages/ai/scripts/tasks.js).

packages/server/build-requirements.txt is restored byte-for-byte to its previous
state. The 'rocketride[otel]' self-reference in the dev/test extras stays — that
is the source-checkout path — and the hard imports in test_otel_mapper.py /
test_otel_setup.py stay, so a missing dependency still fails collection instead
of skipping. The tests no longer depend on the crewai node pulling opentelemetry
in transitively either way.

Docstrings in both test modules and the extras comment in pyproject.toml now
point at the new file and step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(client-python): surface the OTel bridge on the Python SDK overview

Every other page under docs/public/python/ has a card in the SDK overview's
"Explore the SDK" grid; the newly re-homed otel-bridge page did not, so it
was reachable only from the sidebar and the CLI reference.

Adds the card last in the grid, matching sidebar_position: 13. Uses LuRadar,
which is not already taken by another card in this file and is present in the
Lucide set react-icons v5 ships.

This replaces the "OpenTelemetry export (`rocketride otel`)" section that used
to live in docs/README-python-client.md and packages/client-python/docs/index.md
— both deleted by the docs revamp (#2139).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(clients): declare otel-bridge as a Python-only site page

The client-docs gate requires both folders to publish the same page set, with
single-language extras declared in two places: PY_ONLY_PAGES/TS_ONLY_PAGES in
scripts/validate-client-docs.py, and the Site page set section of the schema.
docs/public/python/otel-bridge.md was added without either, so 'Docs site build'
failed with: python: undeclared site page otel-bridge.md.

The page is genuinely Python-only — the TypeScript SDK ships no OTel bridge
(no otel surface under packages/client-typescript/src, no page in
docs/public/typescript/) — so it belongs in PY_ONLY_PAGES rather than being
mirrored. Declared at sidebar_position 13, matching the TypeScript-only
database-sequelize extra.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(sdk): name the private SDK attribute the OTel bridge hooks

The bridge receives monitor events by assigning its dispatcher to
RocketRideClient._caller_on_event — the attribute EventMixin.__init__
creates from the public on_event= kwarg and EventMixin.on_event awaits
for every DAP envelope. It assigns rather than passing on_event= because
it is handed an already-built client and must restore the previous
handler on exit.

Nothing in the SDK's public surface pins that name, and a rename would
fail silently: assigning an unknown attribute to a Python object
succeeds, so the bridge would keep connecting, keep subscribing, keep
logging "reconnected", and never export another span.

Lift the string into CLIENT_EVENT_HOOK with the rationale next to it, so
the coupling is a single named seam a test can assert on instead of a
literal repeated across three call sites. No behaviour change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(sdk): pin the OTel bridge against the real RocketRideClient

The otel suite was mock-only past the mapper: test_otel_bridge.py and
test_otel_cli.py each define their own FakeClient, and the provider tests
monkeypatch build_providers, FlowSpanMapper and MetricsMapper out. That
covers the run loop's branches but leaves the bridge's one point of
coupling to the SDK untested — and a fake that defines
_caller_on_event itself can never notice the SDK dropping it.

Measured: renaming _caller_on_event to _caller_on_event_renamed across
mixins/events.py and mixins/data.py — a change that stops the bridge
exporting anything at all — leaves all 141 existing otel tests green.

Add tests/test_otel_client_contract.py, which constructs a REAL
RocketRideClient and stubs only the transport (call/is_connected); the
SDK's add_monitor bookkeeping, EventMixin.on_event dispatch, the mappers
and the OpenTelemetry provider pipeline all run for real:

  - three tests pinning CLIENT_EVENT_HOOK from different angles: the
    attribute exists in vars() on a real client, it is where the public
    on_event= kwarg lands, and the SDK's dispatch actually awaits it
  - run_bridge end to end on a real client, with events pushed through
    EventMixin.on_event rather than a fake's emit(), asserting routing,
    the unconsumed-event case, and hook restoration on shutdown
  - the wildcard subscription as the real SDK encodes it on the wire,
    plus the _monitor_keys entry the bridge relies on for replay after a
    reconnect
  - the 24-record wire fixture replayed to real exported spans, matching
    the 10-span forest test_otel_mapper.py gets by feeding the mappers
    directly, so the bridge provably adds, drops and reorders nothing

Under the same simulated rename, six of these seven fail.

Also add an opt-in live export against a real OTLP collector, gated on
ROCKETRIDE_OTEL_TEST_ENDPOINT via the same skipif shape
RocketRideClient_test.py uses for requires_llm. It starts no collector
and changes no CI step; without the env var it simply skips.

Note the new file in the client-python:setup-test-deps comment (it
imports the OTel SDK at collection time) and cross-reference it from
test_otel_bridge.py's docstring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: nihalnihalani <24360630+nihalnihalani@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
AviNormie pushed a commit to AviNormie/rocketride-server that referenced this pull request Nov 9, 2026
…mented ingester protocol (zero engine changes) (rocketride-org#1612)

* feat(sdk): rocketride.otelbridge — OpenTelemetry export over the documented ingester protocol (rocketride-org#1609)

Maps the engine's monitor event stream to standard OTel telemetry with
zero engine/server changes — the client-side refinement invited by
tracking issue rocketride-org#616 after the engine-invasive rocketride-org#498 was archived.

- mapper.py: stateful FlowSpanMapper reconstructs span trees from
  apaevt_flow — task root span, pipe span per pipe id, component child
  spans paired by component identity (per the observability doc's
  warning against stack-position pairing); trace.error -> span status
  ERROR + exception event; apaevt_sse -> span events; unclosed spans at
  end/shutdown flagged rocketride.span.unclosed=true; per-pipe state is
  bounded. MetricsMapper maps apaevt_status_update snapshots to OTel
  instruments (object counts, rates, cpu/gpu memory + peaks).
- Semantic conventions per the July 2026 semantic-conventions-genai
  snapshot: gen_ai.provider.name (not the deprecated gen_ai.system),
  gen_ai.operation.name from the well-known list only, span names
  '{operation} {model}' style, INTERNAL span kind for in-process
  component executions.
- Privacy by default: lane payloads / prompt / result content never
  reach attributes or events unless include_content is set (8KB
  truncation cap when it is); enforced by a sentinel-scan test across
  all exported attributes and events.
- setup.py: TracerProvider/MeterProvider + OTLP http/protobuf exporters;
  ALL opentelemetry imports are lazy so the base package imports without
  the extra.

53 unit tests (InMemorySpanExporter/InMemoryMetricReader; importorskip
so the base CI matrix stays green), including a full replay of the
captured real-engine event fixture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(cli): add 'rocketride otel' subcommand + optional [otel] extra (rocketride-org#1609)

rocketride otel [--endpoint URL] [--protocol http|grpc] [--service-name]
[--headers k=v,...] [--include-content] [--no-metrics] + the standard
connection args. Subscribes via rrext_monitor with token '*' (the
observability doc's recommended ingestion scope, types TASK|SUMMARY|
FLOW|SSE), dispatches to the mappers, reconnects with capped backoff
and resubscribes (subscriptions are per-connection and not durable),
and flushes exporters on SIGINT/SIGTERM.

Endpoint resolution honors the standard OTLP env contract: explicit
--endpoint > OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (verbatim) >
OTEL_EXPORTER_OTLP_ENDPOINT (base URL + /v1/traces|/v1/metrics) >
default. Packaging: opentelemetry-sdk + OTLP http exporter live behind
an optional extra — the base install gains zero dependencies, and
running the subcommand without the extra exits 2 with the exact
"pip install 'rocketride[otel]'" hint before any connection attempt.

25 tests: config precedence matrix, dispatch/reconnect/resubscribe with
a fake client, shutdown flush ordering, missing-extra exit path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: OpenTelemetry bridge guide with Jaeger, Langfuse, and Datadog recipes

Setup guide covering the span model (task -> pipe -> component tree),
the privacy default and --include-content, the FLOW-requires-trace-level
callout, reconnect semantics, and copy-paste recipes with exact OTLP
endpoints/auth for Jaeger all-in-one, Langfuse (/api/public/otel, Basic
auth), and Datadog. Notes that Jaeger does not ingest OTLP metrics —
pair it with --no-metrics or point metrics at a collector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sdk): handle partial OpenTelemetry installs — guard the HTTP exporter import

CI exposed a real environment our matrix missed: opentelemetry-api/sdk
present as another package's transitive dependency (tool_daytona pulls
them in) while the OTLP HTTP exporter is absent. Two fixes:

- setup.py: the http trace/metric exporter imports are now guarded like
  the grpc ones — a partial install raises OtelNotInstalledError with
  the pip install 'rocketride[otel]' hint (CLI exit 2) instead of a raw
  ModuleNotFoundError. Verified live in a reproduced partial venv.
- test_otel_setup.py: importorskip targets the most specific module the
  tests exercise (opentelemetry.exporter.otlp.proto.http), so the module
  skips cleanly in CI's partial environment; plus a regression test that
  simulates the partial install via a None sys.modules entry and asserts
  the friendly error.

Partial venv: 63 passed + 1 module skip; full extra: 79 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sdk): OTel bridge review round — privacy-gate errors, run-id promotion, header precedence, shutdown ordering

CodeRabbit review round on rocketride-org#1612, each finding verified before acting:

- mapper: trace.error no longer bypasses the privacy gate — by default
  only the error signal exports (status ERROR, bare exception event,
  generic description); --include-content restores verbatim text, now
  subject to the same 8KB truncation strings previously skipped. The
  sentinel privacy test plants the sentinel in trace.error.
- mapper: a run first seen via flow events (before the task 'running'
  snapshot) is now promoted from its project|source fallback key to the
  canonical run id when it arrives — previously a second run state was
  created (broken parentage) and the orphaned fallback state leaked
  permanently. Confirmed with a failing repro; guard test ensures two
  distinct canonical runs sharing (project, source) never merge.
- config/setup: only --headers become explicit exporter headers; env
  headers are left to the SDK so OTEL_EXPORTER_OTLP_TRACES_HEADERS /
  _METRICS_HEADERS signal precedence works (it was silently defeated).
  Precedence matrix tests added.
- setup: meter provider shutdown wrapped in finally so metrics flush
  even when tracer shutdown raises.
- CLI: --trace-level none message now says flow tracing stays disabled;
  docs recipes pass collector auth via OTEL_EXPORTER_OTLP_HEADERS env
  instead of argv (shell-history hygiene); error-mapping doc wording
  matches the implementation (component leave events).

Full extra: 90 passed. Partial install (api/sdk without exporter):
69 passed + clean skip. ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sdk): absorb stranded fallback run state when canonical state precedes it

Follow-on to the run-promotion fix: an SSE event creates the canonical
run state WITHOUT registering the (project, source) alias (SSE bodies
carry no project/source per the protocol), so an id-less flow for the
same run could still open a parallel fallback-keyed state — and because
the canonical state already existed, the promotion branch was skipped,
stranding the fallback state (kept alive forever by snapshot
reconciliation) with duplicate spans.

_get_run now reconciles whenever an event carries both the canonical id
and project/source: no canonical state → promote (unchanged); canonical
state exists → _absorb_fallback_run migrates non-colliding pipes (open
spans re-stamped with the canonical run id), closes colliding pipes as
rocketride.span.unclosed=true, and drops the fallback key. Only
fallback-format keys are touched, so distinct canonical runs never
merge (guard test unchanged and green).

Both new regression tests failed before the fix. Full extra: 92 passed;
partial install: 71 passed + expected skip; ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(sdk): stamp OTel spans with the engine's body.eventTime

The bridge was written against a monitor protocol that carried no event
timestamps, so every span used the bridge's own clock at event arrival and
the guide documented that as a limitation. Since rocketride-org#1661 the engine stamps
`body.eventTime` (epoch seconds, set once at ingress by the run-log
continuum) on every forwarded event, and derived flow events inherit their
source event's stamp — so the limitation is no longer true and the
documentation shipped with it was wrong against develop.

Span start/end and span-event timestamps now come from `body.eventTime`,
making durations engine-measured instead of bridge-observed: WebSocket
transport latency and event-loop scheduling no longer inflate every
component span.

The fallback is exactly the previous behaviour. A body with no usable
`eventTime` (an older engine, or a non-float/NaN/non-positive value) and
every span closed outside an event — shutdown, seeded-snapshot
reconciliation, tracked-run cap eviction — use `time.time_ns()`, the same
clock the SDK would have used. Because a run can therefore mix both
sources, each open span records its own start and a close is clamped to
never precede it, so no span can report a negative duration.

Metric points are unchanged: OpenTelemetry metric instruments take no
explicit timestamp, so they remain arrival-timed. That is now stated
rather than left implicit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(client-python): publish the OTel bridge guide and mirror its CLI entry

Two co-location gaps (AGENTS.md:5-28) and one stale claim.

The 369-line guide lived at docs/README-otel-bridge.md, and root docs/ is
not in the docs-site glob (packages/docs/scripts/lib/gather.js:25), so the
guide was GitHub-only. Moved to packages/client-python/docs/otel-bridge.md,
the tree already mounted at develop/python (packages/client-python/scripts/
tasks.js:236) alongside analytics.md and log.md, so it is gathered to the
site at /develop/python/otel-bridge. Its in-repo links become site routes
to match the sibling pages; the three references to it were repointed.

packages/client-python/docs/index.md is the site's Python page and mirrors
the PyPI README's ## CLI block line for line, but had no `rocketride otel`
row and none of the OpenTelemetry paragraph the README gained. Added both.

The guide's ## Limitations still said span timestamps are arrival time
because the wire carries none; the previous commit made spans engine-timed,
so that bullet now describes what actually remains bridge-timed (spans
closed outside an event, pre-continuum engines, and all metric points).

The Python blocks in the guide are also reformatted: ruff formats Markdown
code blocks as of 0.16, and CI's ruff-action is unpinned, so the file failed
`ruff format --check` today at either path. Repo-wide check is clean again.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(changelog): record the rocketride otel bridge under [Unreleased]

CHANGELOG.md [Unreleased] ### Added is the convention for user-visible
additions and is kept alphabetical by scope; a new CLI command and a new
optional extra qualify.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(client-python): name Grafana Tempo in the OTel backend recipes

Issue rocketride-org#1609's acceptance list says "Jaeger/Tempo", and the guide covered
Tempo only implicitly as "Grafana stacks". Tempo is the OTLP receiver a
Grafana user actually points the bridge at, and it needs one non-obvious
step (enable distributor.receivers.otlp) plus a separate metrics backend,
so it is worth naming rather than leaving a reader to infer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sdk): address CodeRabbit review round on the rebased head

Nine threads on the rebased head, all in the OTel bridge. Dispositions:

- docs/otel-bridge.md:151 (Sensitive Data Exposure, CWE-522) — FIXED. The
  Langfuse and LangSmith recipes now prompt for the keys (printf + read -rs,
  spelled identically in bash and zsh) instead of embedding placeholders that
  operators replace with live secrets in a command line, and point at a secret
  manager as the better source. The base64 pipeline is now
  `printf '%s' ... | base64 | tr -d '\r\n'`: real Langfuse keys make the
  joined string 85 characters, past GNU base64's 76-character wrap, and
  command substitution strips only trailing newlines, so the embedded break
  would have produced an invalid Authorization header.

- cli/commands/otel.py:207 (Sensitive Data Exposure, CWE-532) — FIXED. The
  startup line ran `config.endpoint` straight to stdout; an endpoint can carry
  userinfo credentials or a signed query string. It is now always passed
  through `redact_endpoint()`, which drops userinfo, query and fragment while
  keeping scheme/host/port and the ingest path operators actually verify.

- cli/main.py:466, otelbridge/config.py:177, otelbridge/setup.py:123
  (Cleartext Transmission, CWE-319) — FIXED, once, in one place.
  `validate_transport_security()` runs before any exporter exists (in
  `build_providers`, and again in the CLI so the failure is exit code 2 rather
  than a traceback): a credential-looking header name from `--headers` or any
  of the three OTEL_EXPORTER_OTLP_*HEADERS variables, plus a non-loopback
  cleartext endpoint for either signal, aborts startup. Header VALUES are
  never read or echoed — only names reach the message. Loopback stays exempt
  for local collectors; `--insecure` / ROCKETRIDE_OTEL_ALLOW_INSECURE=1 is the
  explicit opt-out and warns on stderr.

- otelbridge/config.py:189 (CWE-201, redirect header forwarding) — FIXED. The
  OTLP/HTTP exporters now get a session that never follows redirects.
  `requests` strips Authorization only when the redirect changes host, so a
  3xx replayed x-api-key to the target and even kept Authorization on an
  https -> http downgrade to the same host.

- otelbridge/config.py:177 (endpoint precedence) — FIXED. `from_args_env` no
  longer pre-reads OTEL_EXPORTER_OTLP_ENDPOINT into an explicit exporter
  endpoint: doing so overrode the signal-specific _TRACES_ENDPOINT /
  _METRICS_ENDPOINT, inverting the OTLP spec order — the same defect already
  fixed for the header variables in c23b312. The SDK now resolves all three,
  and `effective_endpoint()` mirrors its order read-only for the startup line
  and the security check.

- otelbridge/bridge.py:207 (signal ownership) — FIXED. `loop.add_signal_handler`
  replaces rather than chains, and asyncio exposes no way to restore the
  displaced callback, so the bridge no longer touches a signal the running
  loop already handles; `install_signal_handlers=False` turns registration off
  entirely for embedders.

- otelbridge/mapper.py:611 (unbounded run.pipes) — FIXED. MAX_TRACKED_RUNS did
  not bound pipes: a run that keeps receiving events is re-inserted as
  most-recently-eventful and never evicted, while every begin (or event for a
  never-begun pipe) adds a _PipeState that only a matching end removes, and a
  reconnect does not replay missed ends. MAX_TRACKED_PIPES_PER_RUN now closes
  the least-recently-eventful pipe as unclosed, with recency updated on pipe
  activity rather than inferred from insertion order.

- otelbridge/setup.py:202 (tracer leak) — FIXED. A failure anywhere in the
  metric half left the BatchSpanProcessor worker running with no handle to
  stop it, because `shutdown` had not been created yet. That path now shuts
  the tracer provider down before re-raising.

Tests: 32 added (131 in tests/test_otel_*.py, was 99), covering stdout
redaction, the refusal and its opt-out, env-var and signal-specific endpoint
coverage, the non-redirecting session, the tracer teardown, signal ownership
and the pipe cap's LRU behaviour.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(sdk): skip the loop signal-handler ownership test on Windows

asyncio's ProactorEventLoop raises NotImplementedError from
add_signal_handler, so the test cannot set up its precondition there.
The bridge itself already tolerates that platform (the other signal
tests pass on Windows); only the test needs the platform guard.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(cli): document the otel --insecure flag in the site CLI reference

The `otel` command grew an `--insecure` opt-out for the cleartext-credential
guard, but the docs-site "Key flags" table was never updated to match. Users
hitting exit code 2 from `rocketride otel --endpoint http://tempo:4318
--headers Authorization=...` had no way to learn why from the CLI reference —
the flag was only documented in the client-python OTel bridge guide.

Add the missing row, naming the `ROCKETRIDE_OTEL_ALLOW_INSECURE` env var
alongside the flag so the site table matches otel-bridge.md and the argparse
help text.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sdk): declare the otel test dependency and stop orphaning bridge providers

Two review findings on the `rocketride otel` bridge.

1. The opentelemetry dependency of the bridge tests was inherited, not
   declared. tests/test_otel_mapper.py and tests/test_otel_setup.py opened
   with pytest.importorskip('opentelemetry'), and nothing in the test path
   installs the 'otel' extra — the packages reached the engine interpreter
   only as a transitive dependency of the crewai node. Dropping or repinning
   that node would have turned ~1450 lines of mapper/setup coverage into a
   silent skip with CI still green.

   CI never pip-installs the package (pytest runs on the engine interpreter
   importing the package from the source checkout via tests/conftest.py), so the extras alone cannot fix this. The
   deps are added to packages/server/build-requirements.txt, which
   server:setup-test-deps installs through depends() under the merged
   constraints — the one install step that path goes through — and the
   'dev'/'test' extras now pull 'rocketride[otel]' so a source checkout
   matches. Both importorskip guards become hard imports: a missing
   dependency now fails collection instead of skipping.

2. build_providers() always built both provider halves and adopted its
   shutdown only when shutdown_fn was None, so an embedder passing its own
   shutdown_fn left a TracerProvider and its BatchSpanProcessor worker
   thread with nothing to stop them — and the half the caller had already
   supplied via a factory was built, wired and discarded.

   build_providers() takes with_tracer / with_meter (defaulting to the old
   behaviour) and run_bridge asks only for the half it is missing; a
   caller-supplied shutdown_fn is now chained in front of the providers'
   shutdown — caller first, providers in a finally — instead of replacing
   it. The CLI path passes neither argument and is unaffected.

Regression tests cover chain order, the provider shutdown surviving a
raising caller, and which half is built for each factory combination;
test_otel_bridge.py stays runnable without the extra. The embedded-use
ownership rules are documented in the bridge doc.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* build(client-python): install the otel test deps in the test step, not the engine

Review follow-up: the previous commit declared opentelemetry-sdk and
opentelemetry-exporter-otlp-proto-http in packages/server/build-requirements.txt.
That file is installed by server:setup-test-deps on every engine build, so every
engine install carried the OTel SDK and the OTLP exporter whether or not anyone
ran the bridge — which made the summary's "zero new base dependencies" true of
the package but not of the engine.

Move the cost to the step that needs it. A new client-python:setup-test-deps
action installs packages/client-python/tests/requirements.txt into the engine
interpreter through depends(), and runs only inside client-python:test, between
the parallel build group and the test-server bracket:

  server:build -> [nodes:build | ai:build | client-python:build]
               -> client-python:setup-test-deps
               -> py-test-server:setup -> client-python:run-pytest -> teardown

That ordering puts it after the node/ai requirement files land in the dist, so
depends() resolves against the complete constraint set, and before the bracket,
so a cold install does not hold the test server open. This mirrors the existing
packages/ai/tests/requirements.txt pattern (packages/ai/scripts/tasks.js).

packages/server/build-requirements.txt is restored byte-for-byte to its previous
state. The 'rocketride[otel]' self-reference in the dev/test extras stays — that
is the source-checkout path — and the hard imports in test_otel_mapper.py /
test_otel_setup.py stay, so a missing dependency still fails collection instead
of skipping. The tests no longer depend on the crewai node pulling opentelemetry
in transitively either way.

Docstrings in both test modules and the extras comment in pyproject.toml now
point at the new file and step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(client-python): surface the OTel bridge on the Python SDK overview

Every other page under docs/public/python/ has a card in the SDK overview's
"Explore the SDK" grid; the newly re-homed otel-bridge page did not, so it
was reachable only from the sidebar and the CLI reference.

Adds the card last in the grid, matching sidebar_position: 13. Uses LuRadar,
which is not already taken by another card in this file and is present in the
Lucide set react-icons v5 ships.

This replaces the "OpenTelemetry export (`rocketride otel`)" section that used
to live in docs/README-python-client.md and packages/client-python/docs/index.md
— both deleted by the docs revamp (rocketride-org#2139).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(clients): declare otel-bridge as a Python-only site page

The client-docs gate requires both folders to publish the same page set, with
single-language extras declared in two places: PY_ONLY_PAGES/TS_ONLY_PAGES in
scripts/validate-client-docs.py, and the Site page set section of the schema.
docs/public/python/otel-bridge.md was added without either, so 'Docs site build'
failed with: python: undeclared site page otel-bridge.md.

The page is genuinely Python-only — the TypeScript SDK ships no OTel bridge
(no otel surface under packages/client-typescript/src, no page in
docs/public/typescript/) — so it belongs in PY_ONLY_PAGES rather than being
mirrored. Declared at sidebar_position 13, matching the TypeScript-only
database-sequelize extra.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(sdk): name the private SDK attribute the OTel bridge hooks

The bridge receives monitor events by assigning its dispatcher to
RocketRideClient._caller_on_event — the attribute EventMixin.__init__
creates from the public on_event= kwarg and EventMixin.on_event awaits
for every DAP envelope. It assigns rather than passing on_event= because
it is handed an already-built client and must restore the previous
handler on exit.

Nothing in the SDK's public surface pins that name, and a rename would
fail silently: assigning an unknown attribute to a Python object
succeeds, so the bridge would keep connecting, keep subscribing, keep
logging "reconnected", and never export another span.

Lift the string into CLIENT_EVENT_HOOK with the rationale next to it, so
the coupling is a single named seam a test can assert on instead of a
literal repeated across three call sites. No behaviour change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(sdk): pin the OTel bridge against the real RocketRideClient

The otel suite was mock-only past the mapper: test_otel_bridge.py and
test_otel_cli.py each define their own FakeClient, and the provider tests
monkeypatch build_providers, FlowSpanMapper and MetricsMapper out. That
covers the run loop's branches but leaves the bridge's one point of
coupling to the SDK untested — and a fake that defines
_caller_on_event itself can never notice the SDK dropping it.

Measured: renaming _caller_on_event to _caller_on_event_renamed across
mixins/events.py and mixins/data.py — a change that stops the bridge
exporting anything at all — leaves all 141 existing otel tests green.

Add tests/test_otel_client_contract.py, which constructs a REAL
RocketRideClient and stubs only the transport (call/is_connected); the
SDK's add_monitor bookkeeping, EventMixin.on_event dispatch, the mappers
and the OpenTelemetry provider pipeline all run for real:

  - three tests pinning CLIENT_EVENT_HOOK from different angles: the
    attribute exists in vars() on a real client, it is where the public
    on_event= kwarg lands, and the SDK's dispatch actually awaits it
  - run_bridge end to end on a real client, with events pushed through
    EventMixin.on_event rather than a fake's emit(), asserting routing,
    the unconsumed-event case, and hook restoration on shutdown
  - the wildcard subscription as the real SDK encodes it on the wire,
    plus the _monitor_keys entry the bridge relies on for replay after a
    reconnect
  - the 24-record wire fixture replayed to real exported spans, matching
    the 10-span forest test_otel_mapper.py gets by feeding the mappers
    directly, so the bridge provably adds, drops and reorders nothing

Under the same simulated rename, six of these seven fail.

Also add an opt-in live export against a real OTLP collector, gated on
ROCKETRIDE_OTEL_TEST_ENDPOINT via the same skipif shape
RocketRideClient_test.py uses for requires_llm. It starts no collector
and changes no CI step; without the env var it simply skips.

Note the new file in the client-python:setup-test-deps comment (it
imports the OTel SDK at collection time) and cross-reference it from
test_otel_bridge.py's docstring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: nihalnihalani <24360630+nihalnihalani@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hackathon Hackathon project or submission module:ai AI/ML modules zz: good-concept Deprecated 2026-08-05, do not use; reviewer sentiment belongs in comments zz: needs-refinement Deprecated 2026-08-05, do not use; refinement asks belong in comments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants