Repository navigation
feat(engine): add OpenTelemetry tracing and Prometheus metrics - #498
nihalnihalani wants to merge 6 commits into
Conversation
- Add Prometheus metrics: pipeline executions, LLM requests/tokens, vectordb ops, HTTP requests - Add OpenTelemetry tracing with configurable OTLP exporter - Add metrics middleware for automatic HTTP request instrumentation - Add /metrics endpoint for Prometheus scraping - Add comprehensive tests for all observability components (27 tests) - Environment-driven configuration (OTEL_EXPORTER_OTLP_ENDPOINT, ROCKETRIDE_METRICS_PUBLIC, etc.) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds Prometheus metrics, a metrics exposition endpoint, request-observability middleware, and OpenTelemetry tracing bootstrap/shutdown; integrates middleware and metrics route into the server and includes end-to-end tests for metrics, middleware, and tracing. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Middleware as MetricsMiddleware
participant App as FastAPI/AppHandler
participant Prom as Prometheus\r\nRegistry
participant OTel as OpenTelemetry\r\nTracer
Client->>Middleware: HTTP request
activate Middleware
Middleware->>OTel: Start Span ("METHOD /normalized/path")\nattrs: http.method, http.url, http.route
activate OTel
Middleware->>App: call_next(request)
activate App
App-->>Middleware: Response (status) or Exception
deactivate App
Middleware->>Prom: Increment HTTP_REQUESTS (method, endpoint, status)
Middleware->>Prom: Observe HTTP_REQUEST_DURATION (method, endpoint, duration)
Middleware->>OTel: Set span attribute http.status_code
OTel-->>Middleware: End Span
deactivate OTel
Middleware-->>Client: Return Response or propagate Exception
deactivate Middleware
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/ai/src/ai/web/endpoints/metrics_endpoint.py`:
- Around line 31-35: The server bootstrap never registers the new metrics
endpoint, so /metrics is unreachable; in the startup code that calls
WebServer.add_route for endpoints (the block that currently adds '/ping',
'/use', '/shutdown', '/status', '/version'), add a call to
WebServer.add_route('/metrics', metrics_endpoint, ['GET'],
public=is_metrics_public()) and ensure metrics_endpoint and is_metrics_public()
are imported/available in that module so the route and its access toggle are
wired up at server initialization.
- Around line 47-53: The is_metrics_public() function currently treats an unset
ROCKETRIDE_METRICS_PUBLIC as true; change it so the default is private by making
the default value 'false' (or otherwise treat missing env var as false) and keep
accepting truthy strings like 'true','1','yes' to opt-in; update the function
body and its docstring accordingly and ensure the function name is unchanged so
route registration logic still calls is_metrics_public().
In `@packages/ai/src/ai/web/metrics/middleware.py`:
- Around line 29-33: WebServer.__init__ currently only registers AuthMiddleware
and CORSMiddleware so MetricsMiddleware never runs; update WebServer.__init__ to
call app.add_middleware(MetricsMiddleware) (registered after AuthMiddleware) so
the request path is resolved and HTTP metrics are recorded; locate the
middleware registration block where AuthMiddleware and CORSMiddleware are added
and append the MetricsMiddleware registration using the same app instance.
- Around line 51-52: Use the matched route pattern from request.scope['route']
instead of request.url.path for the endpoint label to avoid cardinality
explosion; in the middleware (around where method = request.method and path =
request.url.path and the other blocks at lines ~58-64 and ~74-77) retrieve route
= request.scope.get('route') and use route.path (or route.path_format) as the
endpoint template with a fallback to request.url.path when route is missing, and
replace all uses that set the endpoint/time-series label so parameterized routes
emit the template rather than concrete IDs.
In `@packages/ai/src/ai/web/metrics/requirements.txt`:
- Around line 1-5: The new observability deps listed in
packages/ai/src/ai/web/metrics/requirements.txt (prometheus_client,
opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-fastapi,
opentelemetry-exporter-otlp) are not declared in the package's primary install
manifest; add these exact package names (with appropriate version pins, e.g.,
>=0.20.0 / >=1.24.0 / >=0.45b0 as in the requirements file) to the package's
main packaging configuration (pyproject.toml [tool.poetry.dependencies] or
[project.dependencies], or setup.py install_requires) for the ai package so
imports succeed at runtime and CI installs include them.
In `@packages/ai/src/ai/web/metrics/tracing.py`:
- Around line 54-93: The tracing setup is never invoked because
setup_tracing(app) and shutdown_tracing() are not called from the FastAPI
lifespan handlers; modify the server lifecycle wiring to call
setup_tracing(self.app) during startup (inside the existing _on_startup /
startup callback flow) and call shutdown_tracing() during shutdown (inside
_on_shutdown / shutdown callback flow) so the TracerProvider in setup_tracing
and its FastAPI instrumentation are activated in production; look for
setup_tracing and shutdown_tracing and the methods _on_startup and _on_shutdown
in server.py to add these calls.
- Around line 65-85: The code currently calls
trace.set_tracer_provider(provider) unconditionally which cannot override an
existing global provider; change it to first inspect the current global via
trace.get_tracer_provider() and only call trace.set_tracer_provider(provider)
(and assign _tracer_provider = provider) if the current provider is the
default/no-op instance (or otherwise uninitialized); if a provider is already
set, skip replacing it and optionally log a debug message. Apply the same guard
around the other occurrence referenced (lines ~113-116) so both initialization
sites check trace.get_tracer_provider() before calling
trace.set_tracer_provider().
In `@test/nodes/test_observability.py`:
- Around line 279-287: The test patches tracing.FastAPIInstrumentor but
setup_tracing does a local import from opentelemetry.instrumentation.fastapi
inside the function, so update the test test_setup_tracing_with_fastapi_app to
patch the real symbol opentelemetry.instrumentation.fastapi.FastAPIInstrumentor
(e.g. with patch('opentelemetry.instrumentation.fastapi.FastAPIInstrumentor',
create=True)), set its instrument_app to a MagicMock, call
tracing.setup_tracing(app=mock_app), then assert the mocked instrument_app was
called and shutdown_tracing() as before; reference setup_tracing and
FastAPIInstrumentor to locate where to change the patch and add the assert.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: f578f830-2921-49aa-9c47-d7b75f5d2075
📒 Files selected for processing (6)
packages/ai/src/ai/web/endpoints/metrics_endpoint.pypackages/ai/src/ai/web/metrics/middleware.pypackages/ai/src/ai/web/metrics/prometheus_metrics.pypackages/ai/src/ai/web/metrics/requirements.txtpackages/ai/src/ai/web/metrics/tracing.pytest/nodes/test_observability.py
| prometheus_client>=0.20.0 | ||
| opentelemetry-api>=1.24.0 | ||
| opentelemetry-sdk>=1.24.0 | ||
| opentelemetry-instrumentation-fastapi>=0.45b0 | ||
| opentelemetry-exporter-otlp>=1.24.0 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## References to the new requirements file"
rg -n --hidden "packages/ai/src/ai/web/metrics/requirements\.txt|ai/web/metrics/requirements\.txt" . || true
echo
echo "## Manifest entries for the observability dependencies"
rg -n --hidden \
-g 'pyproject.toml' \
-g 'setup.py' \
-g 'setup.cfg' \
-g 'requirements*.txt' \
-g 'Pipfile' \
-g 'poetry.lock' \
"prometheus_client|opentelemetry-api|opentelemetry-sdk|opentelemetry-exporter-otlp|opentelemetry-instrumentation-fastapi" . || trueRepository: rocketride-org/rocketride-server
Length of output: 594
Add dependencies to the main package manifest.
The new packages/ai/src/ai/web/metrics/requirements.txt is never referenced and the five observability dependencies (prometheus_client, opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-fastapi, opentelemetry-exporter-otlp) are not declared in any main packaging manifest (pyproject.toml, setup.py, or equivalent). These imports will fail at runtime unless the dependencies are added to the package's primary installation configuration.
🧰 Tools
🪛 OSV Scanner (2.3.5)
[HIGH] 1-1: requests 2.9.2: undefined
(PYSEC-2018-28)
[HIGH] 1-1: requests 2.9.2: undefined
(PYSEC-2023-74)
[HIGH] 1-1: requests 2.9.2: Requests vulnerable to .netrc credentials leak via malicious URLs
[HIGH] 1-1: requests 2.9.2: Requests Session object does not verify requests after making first request with verify=False
[HIGH] 1-1: requests 2.9.2: Requests has Insecure Temp File Reuse in its extract_zipped_paths() utility function
[HIGH] 1-1: requests 2.9.2: Unintended leak of Proxy-Authorization header in requests
[HIGH] 1-1: requests 2.9.2: Insufficiently Protected Credentials in Requests
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/ai/src/ai/web/metrics/requirements.txt` around lines 1 - 5, The new
observability deps listed in packages/ai/src/ai/web/metrics/requirements.txt
(prometheus_client, opentelemetry-api, opentelemetry-sdk,
opentelemetry-instrumentation-fastapi, opentelemetry-exporter-otlp) are not
declared in the package's primary install manifest; add these exact package
names (with appropriate version pins, e.g., >=0.20.0 / >=1.24.0 / >=0.45b0 as in
the requirements file) to the package's main packaging configuration
(pyproject.toml [tool.poetry.dependencies] or [project.dependencies], or
setup.py install_requires) for the ai package so imports succeed at runtime and
CI installs include them.
There was a problem hiding this comment.
@coderabbitai Good catch. The new requirements.txt in the metrics package is isolated and not referenced by the main package manifest (pyproject.toml / setup.py / requirements.txt). This means the observability dependencies (prometheus_client, opentelemetry-api, opentelemetry-sdk, etc.) will not be installed when the package is deployed. We will add these dependencies to the main package manifest to ensure they are installed as part of the standard deployment. Thank you for the investigation.
There was a problem hiding this comment.
Rate Limit Exceeded
@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 42 seconds before sending another message.
There was a problem hiding this comment.
Fixed in 1479fa6 — added all five observability dependencies (prometheus_client, opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-fastapi, opentelemetry-exporter-otlp) to packages/ai/src/ai/web/requirements.txt, which is the canonical web package manifest used by the build system.
There was a problem hiding this comment.
Rate Limit Exceeded
@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 12 seconds before sending another message.
- Use global prometheus_client.REGISTRY instead of custom CollectorRegistry so instrumentation libraries merge seamlessly and process/platform collectors are included automatically - Normalize URL paths in middleware to route templates via Starlette route matching, with regex fallback for UUIDs and numeric segments, preventing Prometheus label cardinality explosion - Add custom histogram buckets for LLM request durations (0.1s-120s) tuned for real-world LLM latency distribution - Add threading.Lock around TracerProvider setup for thread safety - Add rate limiting documentation for /metrics endpoint security Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (5)
packages/ai/src/ai/web/endpoints/metrics_endpoint.py (2)
31-35:⚠️ Potential issue | 🔴 Critical
/metricsstill needs server route registration.This integration example is not wired in current
packages/ai/src/ai/web/server.pyroute bootstrap, so the endpoint remains unreachable.🔧 Suggested integration in
packages/ai/src/ai/web/server.py+from ai.web.endpoints.metrics_endpoint import metrics_endpoint, is_metrics_public @@ if register_standard_endpoints: self.add_route('/ping', ping, ['GET'], public=True) self.add_route('/use', use, ['POST']) self.add_route('/shutdown', shutdown, ['POST']) + self.add_route('/metrics', metrics_endpoint, ['GET'], public=is_metrics_public())🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/ai/src/ai/web/endpoints/metrics_endpoint.py` around lines 31 - 35, The /metrics endpoint (metrics_endpoint) is not registered with the server bootstrap so it is unreachable; update the server route registration code that calls WebServer.add_route (or server.add_route) to add the '/metrics' GET route pointing to metrics_endpoint and set the appropriate public/is_public flag; locate the route-registration function (the server startup or route bootstrap that registers other endpoints) and add a route registration for '/metrics' using metrics_endpoint and the GET method so the endpoint becomes reachable.
50-64:⚠️ Potential issue | 🟠 MajorDefault
/metricsvisibility should be private.Using
'true'as default makes internals publicly exposed unless explicitly hardened.🔧 Suggested change
- Controlled by ``ROCKETRIDE_METRICS_PUBLIC`` (default ``true``). - Set to ``false`` to require authentication for scraping. + Controlled by ``ROCKETRIDE_METRICS_PUBLIC`` (default ``false``). + Set to ``true`` to allow unauthenticated scraping. @@ - return os.environ.get('ROCKETRIDE_METRICS_PUBLIC', 'true').lower() in ('true', '1', 'yes') + return os.environ.get('ROCKETRIDE_METRICS_PUBLIC', 'false').lower() in ('true', '1', 'yes')🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/ai/src/ai/web/endpoints/metrics_endpoint.py` around lines 50 - 64, The current default for ROCKETRIDE_METRICS_PUBLIC is 'true', which exposes /metrics publicly; change the default to 'false' in the environment parse expression (replace 'true' with 'false' in the return statement) and update the surrounding docstring/comments to state that the default is private. Ensure any related unit tests or configuration examples that assume a public default are updated to expect a private default and verify that the function reading ROCKETRIDE_METRICS_PUBLIC still accepts 'true','1','yes' as enabling public metrics.packages/ai/src/ai/web/metrics/middleware.py (1)
29-33:⚠️ Potential issue | 🔴 CriticalRegister
MetricsMiddlewareinWebServer.__init__.
packages/ai/src/ai/web/server.pycurrently shows onlyAuthMiddleware/CORSMiddleware; without adding this middleware, HTTP metrics in this file never execute.🔧 Suggested integration in
packages/ai/src/ai/web/server.py+from ai.web.metrics.middleware import MetricsMiddleware @@ self.app.add_middleware(AuthMiddleware) + self.app.add_middleware(MetricsMiddleware)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/ai/src/ai/web/metrics/middleware.py` around lines 29 - 33, Add MetricsMiddleware registration to WebServer.__init__ so HTTP metrics run; inside the WebServer.__init__ method (where AuthMiddleware and CORSMiddleware are added) call app.add_middleware(MetricsMiddleware) and ensure it is added after AuthMiddleware so routes resolve correctly. Reference the MetricsMiddleware class from ai.web.metrics.middleware and the app.add_middleware call in WebServer.__init__; keep the existing AuthMiddleware and CORSMiddleware registrations and insert the MetricsMiddleware addition immediately after the auth middleware registration.packages/ai/src/ai/web/metrics/tracing.py (2)
32-34:⚠️ Potential issue | 🔴 CriticalWire tracing into server lifespan; otherwise this module stays inactive.
setup_tracing()/shutdown_tracing()are documented here, butpackages/ai/src/ai/web/server.pybootstrap/lifespan still doesn’t call them, so tracing won’t run in production.🔧 Suggested integration in
packages/ai/src/ai/web/server.py+from ai.web.metrics.tracing import setup_tracing, shutdown_tracing @@ async def _on_startup(...): + setup_tracing(self.app) ... @@ async def _on_shutdown(...): + shutdown_tracing() ...🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/ai/src/ai/web/metrics/tracing.py` around lines 32 - 34, The tracing module is never wired into the application lifecycle; call setup_tracing(app) during application startup and shutdown_tracing() on application shutdown so the instrumentation activates. Update the FastAPI/bootstrap lifespan in server.py to invoke setup_tracing(app) when the app starts (e.g., in the lifespan context or startup event) and ensure shutdown_tracing() is called on exit, passing the same app/context so the tracing middleware and automatic FastAPI instrumentation are registered and torn down properly.
93-95:⚠️ Potential issue | 🔴 CriticalDon’t rely on re-calling
set_tracer_providerafter shutdown.After
shutdown_tracing()clears_tracer_provider, a latersetup_tracing()creates a new provider and callstrace.set_tracer_provider(...)again. OpenTelemetry won’t replace an already-set global provider, which can leave local and global providers inconsistent.In opentelemetry-python, what is the behavior of trace.set_tracer_provider() when called more than once? Does it replace the existing global provider?🔧 Suggested guard
from opentelemetry import trace @@ - trace.set_tracer_provider(provider) - _tracer_provider = provider + current = trace.get_tracer_provider() + if isinstance(current, TracerProvider): + _tracer_provider = current + else: + trace.set_tracer_provider(provider) + _tracer_provider = providerAlso applies to: 122-125
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/ai/src/ai/web/metrics/tracing.py` around lines 93 - 95, The code calls trace.set_tracer_provider(provider) unconditionally in setup_tracing(), which won't replace an already-set global provider after shutdown; update setup_tracing() to first check trace.get_tracer_provider() and only call trace.set_tracer_provider(provider) if the global provider is the default/noop provider (or not already the same provider), and update shutdown_tracing() to explicitly reset the global provider (e.g., call trace.set_tracer_provider(trace.NoopTracerProvider()) or equivalent) and clear _tracer_provider so subsequent setup_tracing() can safely install a new provider; reference the functions setup_tracing(), shutdown_tracing(), the module-level _tracer_provider, and trace.get_tracer_provider()/trace.set_tracer_provider().
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/ai/src/ai/web/metrics/tracing.py`:
- Around line 72-75: The early return inside the _setup_lock block skips
FastAPIInstrumentor.instrument_app(app) when _tracer_provider already exists;
change the logic in the setup function(s) around the
_setup_lock/_tracer_provider checks so that if an app argument is provided you
still call FastAPIInstrumentor.instrument_app(app) even when _tracer_provider is
not None. Locate the branches that reference _setup_lock and _tracer_provider
(the first block around lines 72–75 and the similar block at 96–101) and modify
them to conditionally perform instrumentation when app is passed rather than
returning immediately, ensuring you only skip provider reinitialization but
always attempt to instrument the FastAPI app.
---
Duplicate comments:
In `@packages/ai/src/ai/web/endpoints/metrics_endpoint.py`:
- Around line 31-35: The /metrics endpoint (metrics_endpoint) is not registered
with the server bootstrap so it is unreachable; update the server route
registration code that calls WebServer.add_route (or server.add_route) to add
the '/metrics' GET route pointing to metrics_endpoint and set the appropriate
public/is_public flag; locate the route-registration function (the server
startup or route bootstrap that registers other endpoints) and add a route
registration for '/metrics' using metrics_endpoint and the GET method so the
endpoint becomes reachable.
- Around line 50-64: The current default for ROCKETRIDE_METRICS_PUBLIC is
'true', which exposes /metrics publicly; change the default to 'false' in the
environment parse expression (replace 'true' with 'false' in the return
statement) and update the surrounding docstring/comments to state that the
default is private. Ensure any related unit tests or configuration examples that
assume a public default are updated to expect a private default and verify that
the function reading ROCKETRIDE_METRICS_PUBLIC still accepts 'true','1','yes' as
enabling public metrics.
In `@packages/ai/src/ai/web/metrics/middleware.py`:
- Around line 29-33: Add MetricsMiddleware registration to WebServer.__init__ so
HTTP metrics run; inside the WebServer.__init__ method (where AuthMiddleware and
CORSMiddleware are added) call app.add_middleware(MetricsMiddleware) and ensure
it is added after AuthMiddleware so routes resolve correctly. Reference the
MetricsMiddleware class from ai.web.metrics.middleware and the
app.add_middleware call in WebServer.__init__; keep the existing AuthMiddleware
and CORSMiddleware registrations and insert the MetricsMiddleware addition
immediately after the auth middleware registration.
In `@packages/ai/src/ai/web/metrics/tracing.py`:
- Around line 32-34: The tracing module is never wired into the application
lifecycle; call setup_tracing(app) during application startup and
shutdown_tracing() on application shutdown so the instrumentation activates.
Update the FastAPI/bootstrap lifespan in server.py to invoke setup_tracing(app)
when the app starts (e.g., in the lifespan context or startup event) and ensure
shutdown_tracing() is called on exit, passing the same app/context so the
tracing middleware and automatic FastAPI instrumentation are registered and torn
down properly.
- Around line 93-95: The code calls trace.set_tracer_provider(provider)
unconditionally in setup_tracing(), which won't replace an already-set global
provider after shutdown; update setup_tracing() to first check
trace.get_tracer_provider() and only call trace.set_tracer_provider(provider) if
the global provider is the default/noop provider (or not already the same
provider), and update shutdown_tracing() to explicitly reset the global provider
(e.g., call trace.set_tracer_provider(trace.NoopTracerProvider()) or equivalent)
and clear _tracer_provider so subsequent setup_tracing() can safely install a
new provider; reference the functions setup_tracing(), shutdown_tracing(), the
module-level _tracer_provider, and
trace.get_tracer_provider()/trace.set_tracer_provider().
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 2aa740ab-d8b6-4553-a4f1-56441e608b9a
📒 Files selected for processing (4)
packages/ai/src/ai/web/endpoints/metrics_endpoint.pypackages/ai/src/ai/web/metrics/middleware.pypackages/ai/src/ai/web/metrics/prometheus_metrics.pypackages/ai/src/ai/web/metrics/tracing.py
- Default ROCKETRIDE_METRICS_PUBLIC to 'false' (secure by default)
- Use request.scope.get('route') for endpoint label in metrics
middleware instead of re-matching routes manually
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…heus integration - Register /metrics endpoint in server bootstrap via add_route - Register MetricsMiddleware in WebServer.__init__ after AuthMiddleware - Wire setup_tracing(app) into _on_startup and shutdown_tracing() into _on_shutdown - Fix tracing early return that skipped FastAPI app instrumentation when provider already existed; add idempotent _otel_instrumented guard - Add observability deps to web/requirements.txt (canonical package manifest) - Fix test_is_metrics_public_default to expect False (secure default) - Fix test mock patch target for FastAPIInstrumentor to match actual import path - Add scope/app attributes to middleware test mocks for _normalise_path compat Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Addressed CodeRabbit Review Comments (commit 1479fa6)1. /metrics route not registered in server bootstrap (metrics_endpoint.py:35) 2. Default metrics to private (metrics_endpoint.py:53) 3. MetricsMiddleware not registered (middleware.py:33) 4. Middleware test mocks missing scope/app (middleware.py:52) 5. Dependencies not in main package manifest (requirements.txt:5) 6. Tracing not wired into FastAPI lifespan (tracing.py:93) 7. set_tracer_provider cannot safely replace global provider (tracing.py:85) 8. Test patches wrong symbol for FastAPIInstrumentor (test_observability.py:287) 9. App instrumentation skipped when provider already exists (tracing.py:75) |
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/ai/src/ai/web/metrics/tracing.py`:
- Around line 123-131: Protect accesses to _tracer_provider in shutdown_tracing
by acquiring the _setup_lock used by setup_tracing: inside the lock check if
_tracer_provider is not None, store it to a local variable and set the global
_tracer_provider to None, then release the lock and call
local_provider.shutdown(); this prevents races with setup_tracing while avoiding
holding the lock during the potentially slow shutdown call.
In `@packages/ai/src/ai/web/requirements.txt`:
- Around line 7-11: Update the OpenTelemetry entries in requirements.txt: bump
opentelemetry-instrumentation-fastapi to 0.61b0 and change opentelemetry-api and
opentelemetry-sdk from open-ended ">=..." to matching exact versions (e.g.,
opentelemetry-api==1.40.0 and opentelemetry-sdk==1.40.0) to ensure compatibility
and prevent version skew between the core API/SDK and the instrumentation
package.
In `@packages/ai/src/ai/web/server.py`:
- Around line 340-344: The call to shutdown_tracing() happens before awaiting
the user-provided shutdown callback (self._user_shutdown), which prevents any
spans emitted during that callback from being exported; modify the shutdown
sequence in the server shutdown routine so that you first await
self._user_shutdown() (if not None) and only then call shutdown_tracing(), i.e.,
move the shutdown_tracing() invocation to after the await of
self._user_shutdown() to allow user shutdown callbacks to use tracing; ensure
you preserve exception handling around the await so the tracer is still shut
down even if the user callback raises.
In `@test/nodes/test_observability.py`:
- Around line 79-142: Tests rely on the global prometheus_client REGISTRY
(prometheus_metrics.REGISTRY) which makes them order-dependent; update the tests
to use an isolated CollectorRegistry per test (or reset/unregister metrics
between tests) and register the metrics (PIPELINE_EXECUTIONS, LLM_REQUESTS,
LLM_TOKENS, VECTORDB_OPERATIONS, ACTIVE_TASKS, HTTP_REQUESTS) against that
registry so each test reads/writes only its own metrics; ensure each test
references REGISTRY.get_sample_value on the isolated registry instance (or
clears prometheus_metrics.REGISTRY) before incrementing to guarantee
deterministic, order-independent assertions.
- Around line 249-272: Tests leave global OpenTelemetry state set across runs;
ensure each test fully resets both the module-level _tracer_provider and the
global trace provider so setup_tracing() can reinitialize. Add an autouse pytest
fixture in test/nodes/test_observability.py that before each test sets
tracing._tracer_provider = None and calls trace.set_tracer_provider(None)
(import trace from opentelemetry.trace), yield, then call
tracing.shutdown_tracing() in teardown; this guarantees setup_tracing() and
shutdown_tracing() behave idempotently and isolates tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: c9aefb2d-6bb7-47e3-96d8-4c8b300f4225
📒 Files selected for processing (6)
packages/ai/src/ai/web/endpoints/metrics_endpoint.pypackages/ai/src/ai/web/metrics/middleware.pypackages/ai/src/ai/web/metrics/tracing.pypackages/ai/src/ai/web/requirements.txtpackages/ai/src/ai/web/server.pytest/nodes/test_observability.py
| prometheus_client>=0.20.0 | ||
| opentelemetry-api>=1.24.0 | ||
| opentelemetry-sdk>=1.24.0 | ||
| opentelemetry-instrumentation-fastapi>=0.45b0 | ||
| opentelemetry-exporter-otlp>=1.24.0 |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
What are the latest stable versions of opentelemetry-api, opentelemetry-sdk, and opentelemetry-instrumentation-fastapi Python packages, and what version compatibility is required between them?
💡 Result:
The latest stable versions are: - opentelemetry-api: 1.40.0 (released 2026-03-04) - opentelemetry-sdk: 1.40.0 (released 2026-03-04) - opentelemetry-instrumentation-fastapi: 0.61b0 (released 2026-03-04, beta status) opentelemetry-instrumentation-fastapi has beta status (PyPI status 4 - Beta) and no stable (1.x) release yet. Version compatibility: - opentelemetry-sdk 1.40.0 requires exactly opentelemetry-api ==1.40.0. - opentelemetry-instrumentation-fastapi 0.61b0 requires opentelemetry-api ~=1.12 (compatible with 1.40.0) and exact versions of related instrumentation packages like opentelemetry-instrumentation-asgi ==0.61b0. - Use matching major.minor versions from the core (opentelemetry-python) and contrib (opentelemetry-python-contrib) releases, which align monthly (e.g., core 1.40.0 with contrib 0.61b0). SDK pins exact API version; instrumentation uses loose ~= for API.
Citations:
- 1: https://pypi.org/project/opentelemetry-api/
- 2: https://pypi.org/project/opentelemetry-sdk/
- 3: https://github.com/open-telemetry/opentelemetry-python
- 4: https://pypi.org/project/opentelemetry-instrumentation-fastapi/
- 5: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/CHANGELOG.md
- 6: https://github.com/open-telemetry/opentelemetry-python-contrib
Update OpenTelemetry instrumentation package version and ensure API/SDK version alignment.
The OpenTelemetry packages require careful version alignment. The instrumentation package (opentelemetry-instrumentation-fastapi>=0.45b0) is outdated—the latest version is 0.61b0 (released 2026-03-04), which aligns with the latest core packages (API/SDK 1.40.0). Additionally, opentelemetry-sdk requires an exact match with opentelemetry-api (e.g., SDK 1.40.0 requires API ==1.40.0), but your current constraints use >=, which could allow incompatible versions.
Update instrumentation to 0.61b0 and consider pinning API/SDK to the same version for stability (e.g., both 1.40.0), or use exact version constraints instead of >= to prevent version skew.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/ai/src/ai/web/requirements.txt` around lines 7 - 11, Update the
OpenTelemetry entries in requirements.txt: bump
opentelemetry-instrumentation-fastapi to 0.61b0 and change opentelemetry-api and
opentelemetry-sdk from open-ended ">=..." to matching exact versions (e.g.,
opentelemetry-api==1.40.0 and opentelemetry-sdk==1.40.0) to ensure compatibility
and prevent version skew between the core API/SDK and the instrumentation
package.
There was a problem hiding this comment.
Fixed in afe6f7f. Added upper-bound version constraints to all OpenTelemetry dependencies (<2.0.0 for stable packages, <1.0.0 for the beta instrumentation package) to prevent breakage from future major version releases.
There was a problem hiding this comment.
Rate Limit Exceeded
@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 26 seconds before sending another message.
| class TestPrometheusMetrics: | ||
| """Validate that Prometheus metric objects are properly defined and usable.""" | ||
|
|
||
| def test_pipeline_executions_counter_increments(self): | ||
| REGISTRY = prometheus_metrics.REGISTRY | ||
| before = REGISTRY.get_sample_value('rocketride_pipeline_executions_total', {'pipeline_name': 'test_pipe', 'status': 'success'}) or 0.0 | ||
| prometheus_metrics.PIPELINE_EXECUTIONS.labels(pipeline_name='test_pipe', status='success').inc() | ||
| after = REGISTRY.get_sample_value('rocketride_pipeline_executions_total', {'pipeline_name': 'test_pipe', 'status': 'success'}) | ||
| assert after == before + 1.0 | ||
|
|
||
| def test_pipeline_executions_counter_error_label(self): | ||
| REGISTRY = prometheus_metrics.REGISTRY | ||
| before = REGISTRY.get_sample_value('rocketride_pipeline_executions_total', {'pipeline_name': 'err_pipe', 'status': 'error'}) or 0.0 | ||
| prometheus_metrics.PIPELINE_EXECUTIONS.labels(pipeline_name='err_pipe', status='error').inc(3) | ||
| after = REGISTRY.get_sample_value('rocketride_pipeline_executions_total', {'pipeline_name': 'err_pipe', 'status': 'error'}) | ||
| assert after == before + 3.0 | ||
|
|
||
| def test_llm_requests_counter_increments(self): | ||
| REGISTRY = prometheus_metrics.REGISTRY | ||
| before = REGISTRY.get_sample_value('rocketride_llm_requests_total', {'provider': 'openai', 'model': 'gpt-4', 'status': 'success'}) or 0.0 | ||
| prometheus_metrics.LLM_REQUESTS.labels(provider='openai', model='gpt-4', status='success').inc() | ||
| after = REGISTRY.get_sample_value('rocketride_llm_requests_total', {'provider': 'openai', 'model': 'gpt-4', 'status': 'success'}) | ||
| assert after == before + 1.0 | ||
|
|
||
| def test_llm_tokens_counter_by_type(self): | ||
| REGISTRY = prometheus_metrics.REGISTRY | ||
| before_in = REGISTRY.get_sample_value('rocketride_llm_tokens_total', {'provider': 'openai', 'model': 'gpt-4', 'token_type': 'input'}) or 0.0 | ||
| before_out = REGISTRY.get_sample_value('rocketride_llm_tokens_total', {'provider': 'openai', 'model': 'gpt-4', 'token_type': 'output'}) or 0.0 | ||
|
|
||
| prometheus_metrics.LLM_TOKENS.labels(provider='openai', model='gpt-4', token_type='input').inc(100) | ||
| prometheus_metrics.LLM_TOKENS.labels(provider='openai', model='gpt-4', token_type='output').inc(50) | ||
|
|
||
| after_in = REGISTRY.get_sample_value('rocketride_llm_tokens_total', {'provider': 'openai', 'model': 'gpt-4', 'token_type': 'input'}) | ||
| after_out = REGISTRY.get_sample_value('rocketride_llm_tokens_total', {'provider': 'openai', 'model': 'gpt-4', 'token_type': 'output'}) | ||
|
|
||
| assert after_in == before_in + 100.0 | ||
| assert after_out == before_out + 50.0 | ||
|
|
||
| def test_vectordb_operations_counter(self): | ||
| REGISTRY = prometheus_metrics.REGISTRY | ||
| before = REGISTRY.get_sample_value('rocketride_vectordb_operations_total', {'provider': 'milvus', 'operation': 'insert'}) or 0.0 | ||
| prometheus_metrics.VECTORDB_OPERATIONS.labels(provider='milvus', operation='insert').inc() | ||
| after = REGISTRY.get_sample_value('rocketride_vectordb_operations_total', {'provider': 'milvus', 'operation': 'insert'}) | ||
| assert after == before + 1.0 | ||
|
|
||
| def test_active_tasks_gauge(self): | ||
| REGISTRY = prometheus_metrics.REGISTRY | ||
| prometheus_metrics.ACTIVE_TASKS.set(0) | ||
| prometheus_metrics.ACTIVE_TASKS.inc() | ||
| prometheus_metrics.ACTIVE_TASKS.inc() | ||
| val = REGISTRY.get_sample_value('rocketride_active_tasks') | ||
| assert val == 2.0 | ||
|
|
||
| prometheus_metrics.ACTIVE_TASKS.dec() | ||
| val = REGISTRY.get_sample_value('rocketride_active_tasks') | ||
| assert val == 1.0 | ||
|
|
||
| def test_http_requests_counter(self): | ||
| REGISTRY = prometheus_metrics.REGISTRY | ||
| before = REGISTRY.get_sample_value('rocketride_http_requests_total', {'method': 'GET', 'endpoint': '/ping', 'status_code': '200'}) or 0.0 | ||
| prometheus_metrics.HTTP_REQUESTS.labels(method='GET', endpoint='/ping', status_code='200').inc() | ||
| after = REGISTRY.get_sample_value('rocketride_http_requests_total', {'method': 'GET', 'endpoint': '/ping', 'status_code': '200'}) | ||
| assert after == before + 1.0 | ||
|
|
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Tests use shared Prometheus REGISTRY without isolation.
The prometheus_client REGISTRY is global and shared across tests. Since tests increment counters and don't reset them, tests depend on execution order and previous test state (using before/after pattern). This works but makes tests order-dependent. Consider using REGISTRY.get_sample_value(...) or 0.0 consistently (which you do) or creating isolated registries per test class for true isolation.
🧰 Tools
🪛 Ruff (0.15.7)
[error] 108-108: Possible hardcoded password assigned to argument: "token_type"
(S106)
[error] 109-109: Possible hardcoded password assigned to argument: "token_type"
(S106)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@test/nodes/test_observability.py` around lines 79 - 142, Tests rely on the
global prometheus_client REGISTRY (prometheus_metrics.REGISTRY) which makes them
order-dependent; update the tests to use an isolated CollectorRegistry per test
(or reset/unregister metrics between tests) and register the metrics
(PIPELINE_EXECUTIONS, LLM_REQUESTS, LLM_TOKENS, VECTORDB_OPERATIONS,
ACTIVE_TASKS, HTTP_REQUESTS) against that registry so each test reads/writes
only its own metrics; ensure each test references REGISTRY.get_sample_value on
the isolated registry instance (or clears prometheus_metrics.REGISTRY) before
incrementing to guarantee deterministic, order-independent assertions.
There was a problem hiding this comment.
Acknowledged. The before/after pattern used throughout these tests already handles the shared REGISTRY correctly — each test measures deltas rather than absolute values, so execution order does not affect correctness. Creating isolated registries per test class would add complexity without meaningful benefit here.
There was a problem hiding this comment.
Rate Limit Exceeded
@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 23 seconds before sending another message.
| def test_setup_tracing_none_exporter(self): | ||
| """With OTEL_EXPORTER_TYPE=none, setup should succeed without network calls.""" | ||
| with patch.dict(os.environ, {'OTEL_EXPORTER_TYPE': 'none', 'OTEL_SERVICE_NAME': 'rocketride-test'}): | ||
| provider = tracing.setup_tracing() | ||
| assert provider is not None | ||
| tracing.shutdown_tracing() | ||
|
|
||
| def test_setup_tracing_console_exporter(self): | ||
| """Console exporter should initialise without errors.""" | ||
| with patch.dict(os.environ, {'OTEL_EXPORTER_TYPE': 'console', 'OTEL_SERVICE_NAME': 'rocketride-test'}): | ||
| provider = tracing.setup_tracing() | ||
| assert provider is not None | ||
| tracing.shutdown_tracing() | ||
|
|
||
| def test_get_tracer_returns_tracer(self): | ||
| """get_tracer() should return a valid Tracer object.""" | ||
| with patch.dict(os.environ, {'OTEL_EXPORTER_TYPE': 'none'}): | ||
| tracing.setup_tracing() | ||
| tracer = tracing.get_tracer('test-module') | ||
| assert tracer is not None | ||
| # Should be able to start a span | ||
| with tracer.start_as_current_span('test-span') as span: | ||
| assert span is not None | ||
| tracing.shutdown_tracing() |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Tests may not properly isolate tracing provider state.
Each test calls setup_tracing() but the module-level _tracer_provider persists across tests. Due to OpenTelemetry's singleton behavior for trace.set_tracer_provider(), once a provider is set globally, subsequent calls in other tests won't replace it. While shutdown_tracing() clears _tracer_provider, it doesn't reset the global trace provider.
Consider adding a fixture that fully resets tracing state between tests, or acknowledge that tests exercise the idempotent behavior rather than isolated initialization.
♻️ Suggested test fixture for isolation
`@pytest.fixture`(autouse=True)
def reset_tracing_state():
"""Reset tracing module state before each test."""
tracing._tracer_provider = None
yield
tracing.shutdown_tracing()🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@test/nodes/test_observability.py` around lines 249 - 272, Tests leave global
OpenTelemetry state set across runs; ensure each test fully resets both the
module-level _tracer_provider and the global trace provider so setup_tracing()
can reinitialize. Add an autouse pytest fixture in
test/nodes/test_observability.py that before each test sets
tracing._tracer_provider = None and calls trace.set_tracer_provider(None)
(import trace from opentelemetry.trace), yield, then call
tracing.shutdown_tracing() in teardown; this guarantees setup_tracing() and
shutdown_tracing() behave idempotently and isolates tests.
There was a problem hiding this comment.
Fixed in afe6f7f. Added an autouse fixture (reset_tracing_state) to TestOpenTelemetryTracing that resets _tracer_provider to None before each test and calls shutdown_tracing() in teardown. This ensures each test starts with a clean module-level state. Note: the global trace.set_tracer_provider() singleton behavior in OpenTelemetry is by design and cannot be fully reset without private API access, but clearing our module reference is sufficient for testing our initialization logic.
There was a problem hiding this comment.
Rate Limit Exceeded
@nihalnihalani have exceeded the limit for the number of chat messages per hour. Please wait 0 minutes and 18 seconds before sending another message.
- Add lock protection to shutdown_tracing() to prevent race conditions with concurrent setup_tracing() calls - Pin OpenTelemetry dependency upper bounds to prevent breaking changes from future major versions - Reorder shutdown_tracing() after user callbacks in server shutdown so user-provided shutdown hooks can still emit spans - Add autouse fixture to reset tracing state between tests for isolation - Add test for concurrent shutdown safety Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Anchor _NUMERIC_RE and _UUID_RE to path segment boundaries to prevent
partial matches like /pipeline/123abc/run -> /pipeline/{id}abc/run
- Add upper bounds to metrics requirements.txt to prevent breaking
changes from major version bumps
- Fix middleware docstring to reflect LIFO middleware ordering (add
before auth, not after)
- Add test for is_metrics_public(True) and path normalisation edge cases
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
packages/ai/src/ai/web/metrics/requirements.txt (1)
1-5: 🧹 Nitpick | 🔵 TrivialThis requirements file appears orphaned and may be dead code.
Based on the context snippets,
packages/ai/src/ai/web/__init__.pyonly loads dependencies viadepends(requirements)pointing to the parentrequirements.txt. This nestedmetrics/requirements.txtis never referenced (no-r metrics/requirements.txtinclude, nodepends()call).Per the past review comments, the author confirmed adding these dependencies to
packages/ai/src/ai/web/requirements.txtinstead. Consider removing this orphaned file to avoid confusion about where dependencies are actually declared.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/ai/src/ai/web/metrics/requirements.txt` around lines 1 - 5, The metrics/requirements.txt file appears orphaned: confirm whether the prometheus/opentelemetry deps were intentionally moved to the parent requirements.txt; if so, delete metrics/requirements.txt to avoid confusion and run a repo-wide search for any references to "metrics/requirements.txt" to ensure nothing breaks; if the file was meant to be included, instead add an explicit include (e.g., -r metrics/requirements.txt) or a depends(...) call from the parent requirements loader so the prometheus_client and opentelemetry entries are actually consumed.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/ai/src/ai/web/metrics/middleware.py`:
- Around line 29-34: The docstring on MetricsMiddleware incorrectly instructs to
"add before the auth middleware" and misstates execution order; update the
docstring in MetricsMiddleware to reflect that Starlette middleware is LIFO and
that if AuthMiddleware is added first and MetricsMiddleware second (as done
where AuthMiddleware then MetricsMiddleware are registered), MetricsMiddleware
will run before AuthMiddleware, so route resolution may not yet be complete when
_normalise_path() runs and auth-failing requests will still be instrumented;
clarify the intended behavior (either instruct to add MetricsMiddleware before
AuthMiddleware to run after auth or explicitly document that MetricsMiddleware
runs before auth and handles missing route via request.scope['route'] fallback)
and reference MetricsMiddleware, AuthMiddleware and _normalise_path() in the
text.
---
Duplicate comments:
In `@packages/ai/src/ai/web/metrics/requirements.txt`:
- Around line 1-5: The metrics/requirements.txt file appears orphaned: confirm
whether the prometheus/opentelemetry deps were intentionally moved to the parent
requirements.txt; if so, delete metrics/requirements.txt to avoid confusion and
run a repo-wide search for any references to "metrics/requirements.txt" to
ensure nothing breaks; if the file was meant to be included, instead add an
explicit include (e.g., -r metrics/requirements.txt) or a depends(...) call from
the parent requirements loader so the prometheus_client and opentelemetry
entries are actually consumed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 33781512-7346-4e72-a19e-a1cc73c042e6
📒 Files selected for processing (3)
packages/ai/src/ai/web/metrics/middleware.pypackages/ai/src/ai/web/metrics/requirements.txttest/nodes/test_observability.py
| Usage — add *before* the auth middleware (Starlette middleware is LIFO, | ||
| so this will execute after auth has resolved the route path):: | ||
|
|
||
| from ai.web.metrics.middleware import MetricsMiddleware | ||
|
|
||
| app.add_middleware(MetricsMiddleware) |
There was a problem hiding this comment.
Docstring contradicts actual middleware registration order.
The docstring says "add before the auth middleware (Starlette middleware is LIFO, so this will execute after auth has resolved the route path)". However, in packages/ai/src/ai/web/server.py lines 135-136, AuthMiddleware is added first, then MetricsMiddleware is added second. With LIFO ordering, this means MetricsMiddleware executes before AuthMiddleware, not after.
This means:
- Route resolution may not have completed when
_normalise_path()runs (therequest.scope['route']fallback handles this) - Requests that fail auth (401/403) are still instrumented, which is actually desirable for observability
Consider updating the docstring to match reality or clarifying the intended behavior.
📝 Suggested docstring fix
-Usage — add *before* the auth middleware (Starlette middleware is LIFO,
-so this will execute after auth has resolved the route path)::
+Usage — add *after* AuthMiddleware in your middleware registration
+(Starlette middleware is LIFO, so MetricsMiddleware will execute first
+and capture all requests including auth failures)::🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/ai/src/ai/web/metrics/middleware.py` around lines 29 - 34, The
docstring on MetricsMiddleware incorrectly instructs to "add before the auth
middleware" and misstates execution order; update the docstring in
MetricsMiddleware to reflect that Starlette middleware is LIFO and that if
AuthMiddleware is added first and MetricsMiddleware second (as done where
AuthMiddleware then MetricsMiddleware are registered), MetricsMiddleware will
run before AuthMiddleware, so route resolution may not yet be complete when
_normalise_path() runs and auth-failing requests will still be instrumented;
clarify the intended behavior (either instruct to add MetricsMiddleware before
AuthMiddleware to run after auth or explicitly document that MetricsMiddleware
runs before auth and handles missing route via request.scope['route'] fallback)
and reference MetricsMiddleware, AuthMiddleware and _normalise_path() in the
text.
|
Thank you for the contribution and the detailed write-up. After review, this PR is not ready for merge in its current state and is being closed to keep the PR list clean. The concept and proposed changes have been archived in #616 for further elaboration and future consideration. |
…mented ingester protocol (rocketride-org#1609) Maps the engine's monitor event stream to standard OTel telemetry with zero engine/server changes — the client-side refinement invited by tracking issue rocketride-org#616 after the engine-invasive rocketride-org#498 was archived. - mapper.py: stateful FlowSpanMapper reconstructs span trees from apaevt_flow — task root span, pipe span per pipe id, component child spans paired by component identity (per the observability doc's warning against stack-position pairing); trace.error -> span status ERROR + exception event; apaevt_sse -> span events; unclosed spans at end/shutdown flagged rocketride.span.unclosed=true; per-pipe state is bounded. MetricsMapper maps apaevt_status_update snapshots to OTel instruments (object counts, rates, cpu/gpu memory + peaks). - Semantic conventions per the July 2026 semantic-conventions-genai snapshot: gen_ai.provider.name (not the deprecated gen_ai.system), gen_ai.operation.name from the well-known list only, span names '{operation} {model}' style, INTERNAL span kind for in-process component executions. - Privacy by default: lane payloads / prompt / result content never reach attributes or events unless include_content is set (8KB truncation cap when it is); enforced by a sentinel-scan test across all exported attributes and events. - setup.py: TracerProvider/MeterProvider + OTLP http/protobuf exporters; ALL opentelemetry imports are lazy so the base package imports without the extra. 53 unit tests (InMemorySpanExporter/InMemoryMetricReader; importorskip so the base CI matrix stays green), including a full replay of the captured real-engine event fixture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mented ingester protocol (zero engine changes) (#1612) * feat(sdk): rocketride.otelbridge — OpenTelemetry export over the documented ingester protocol (#1609) Maps the engine's monitor event stream to standard OTel telemetry with zero engine/server changes — the client-side refinement invited by tracking issue #616 after the engine-invasive #498 was archived. - mapper.py: stateful FlowSpanMapper reconstructs span trees from apaevt_flow — task root span, pipe span per pipe id, component child spans paired by component identity (per the observability doc's warning against stack-position pairing); trace.error -> span status ERROR + exception event; apaevt_sse -> span events; unclosed spans at end/shutdown flagged rocketride.span.unclosed=true; per-pipe state is bounded. MetricsMapper maps apaevt_status_update snapshots to OTel instruments (object counts, rates, cpu/gpu memory + peaks). - Semantic conventions per the July 2026 semantic-conventions-genai snapshot: gen_ai.provider.name (not the deprecated gen_ai.system), gen_ai.operation.name from the well-known list only, span names '{operation} {model}' style, INTERNAL span kind for in-process component executions. - Privacy by default: lane payloads / prompt / result content never reach attributes or events unless include_content is set (8KB truncation cap when it is); enforced by a sentinel-scan test across all exported attributes and events. - setup.py: TracerProvider/MeterProvider + OTLP http/protobuf exporters; ALL opentelemetry imports are lazy so the base package imports without the extra. 53 unit tests (InMemorySpanExporter/InMemoryMetricReader; importorskip so the base CI matrix stays green), including a full replay of the captured real-engine event fixture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(cli): add 'rocketride otel' subcommand + optional [otel] extra (#1609) rocketride otel [--endpoint URL] [--protocol http|grpc] [--service-name] [--headers k=v,...] [--include-content] [--no-metrics] + the standard connection args. Subscribes via rrext_monitor with token '*' (the observability doc's recommended ingestion scope, types TASK|SUMMARY| FLOW|SSE), dispatches to the mappers, reconnects with capped backoff and resubscribes (subscriptions are per-connection and not durable), and flushes exporters on SIGINT/SIGTERM. Endpoint resolution honors the standard OTLP env contract: explicit --endpoint > OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (verbatim) > OTEL_EXPORTER_OTLP_ENDPOINT (base URL + /v1/traces|/v1/metrics) > default. Packaging: opentelemetry-sdk + OTLP http exporter live behind an optional extra — the base install gains zero dependencies, and running the subcommand without the extra exits 2 with the exact "pip install 'rocketride[otel]'" hint before any connection attempt. 25 tests: config precedence matrix, dispatch/reconnect/resubscribe with a fake client, shutdown flush ordering, missing-extra exit path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: OpenTelemetry bridge guide with Jaeger, Langfuse, and Datadog recipes Setup guide covering the span model (task -> pipe -> component tree), the privacy default and --include-content, the FLOW-requires-trace-level callout, reconnect semantics, and copy-paste recipes with exact OTLP endpoints/auth for Jaeger all-in-one, Langfuse (/api/public/otel, Basic auth), and Datadog. Notes that Jaeger does not ingest OTLP metrics — pair it with --no-metrics or point metrics at a collector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): handle partial OpenTelemetry installs — guard the HTTP exporter import CI exposed a real environment our matrix missed: opentelemetry-api/sdk present as another package's transitive dependency (tool_daytona pulls them in) while the OTLP HTTP exporter is absent. Two fixes: - setup.py: the http trace/metric exporter imports are now guarded like the grpc ones — a partial install raises OtelNotInstalledError with the pip install 'rocketride[otel]' hint (CLI exit 2) instead of a raw ModuleNotFoundError. Verified live in a reproduced partial venv. - test_otel_setup.py: importorskip targets the most specific module the tests exercise (opentelemetry.exporter.otlp.proto.http), so the module skips cleanly in CI's partial environment; plus a regression test that simulates the partial install via a None sys.modules entry and asserts the friendly error. Partial venv: 63 passed + 1 module skip; full extra: 79 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): OTel bridge review round — privacy-gate errors, run-id promotion, header precedence, shutdown ordering CodeRabbit review round on #1612, each finding verified before acting: - mapper: trace.error no longer bypasses the privacy gate — by default only the error signal exports (status ERROR, bare exception event, generic description); --include-content restores verbatim text, now subject to the same 8KB truncation strings previously skipped. The sentinel privacy test plants the sentinel in trace.error. - mapper: a run first seen via flow events (before the task 'running' snapshot) is now promoted from its project|source fallback key to the canonical run id when it arrives — previously a second run state was created (broken parentage) and the orphaned fallback state leaked permanently. Confirmed with a failing repro; guard test ensures two distinct canonical runs sharing (project, source) never merge. - config/setup: only --headers become explicit exporter headers; env headers are left to the SDK so OTEL_EXPORTER_OTLP_TRACES_HEADERS / _METRICS_HEADERS signal precedence works (it was silently defeated). Precedence matrix tests added. - setup: meter provider shutdown wrapped in finally so metrics flush even when tracer shutdown raises. - CLI: --trace-level none message now says flow tracing stays disabled; docs recipes pass collector auth via OTEL_EXPORTER_OTLP_HEADERS env instead of argv (shell-history hygiene); error-mapping doc wording matches the implementation (component leave events). Full extra: 90 passed. Partial install (api/sdk without exporter): 69 passed + clean skip. ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): absorb stranded fallback run state when canonical state precedes it Follow-on to the run-promotion fix: an SSE event creates the canonical run state WITHOUT registering the (project, source) alias (SSE bodies carry no project/source per the protocol), so an id-less flow for the same run could still open a parallel fallback-keyed state — and because the canonical state already existed, the promotion branch was skipped, stranding the fallback state (kept alive forever by snapshot reconciliation) with duplicate spans. _get_run now reconciles whenever an event carries both the canonical id and project/source: no canonical state → promote (unchanged); canonical state exists → _absorb_fallback_run migrates non-colliding pipes (open spans re-stamped with the canonical run id), closes colliding pipes as rocketride.span.unclosed=true, and drops the fallback key. Only fallback-format keys are touched, so distinct canonical runs never merge (guard test unchanged and green). Both new regression tests failed before the fix. Full extra: 92 passed; partial install: 71 passed + expected skip; ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): stamp OTel spans with the engine's body.eventTime The bridge was written against a monitor protocol that carried no event timestamps, so every span used the bridge's own clock at event arrival and the guide documented that as a limitation. Since #1661 the engine stamps `body.eventTime` (epoch seconds, set once at ingress by the run-log continuum) on every forwarded event, and derived flow events inherit their source event's stamp — so the limitation is no longer true and the documentation shipped with it was wrong against develop. Span start/end and span-event timestamps now come from `body.eventTime`, making durations engine-measured instead of bridge-observed: WebSocket transport latency and event-loop scheduling no longer inflate every component span. The fallback is exactly the previous behaviour. A body with no usable `eventTime` (an older engine, or a non-float/NaN/non-positive value) and every span closed outside an event — shutdown, seeded-snapshot reconciliation, tracked-run cap eviction — use `time.time_ns()`, the same clock the SDK would have used. Because a run can therefore mix both sources, each open span records its own start and a close is clamped to never precede it, so no span can report a negative duration. Metric points are unchanged: OpenTelemetry metric instruments take no explicit timestamp, so they remain arrival-timed. That is now stated rather than left implicit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(client-python): publish the OTel bridge guide and mirror its CLI entry Two co-location gaps (AGENTS.md:5-28) and one stale claim. The 369-line guide lived at docs/README-otel-bridge.md, and root docs/ is not in the docs-site glob (packages/docs/scripts/lib/gather.js:25), so the guide was GitHub-only. Moved to packages/client-python/docs/otel-bridge.md, the tree already mounted at develop/python (packages/client-python/scripts/ tasks.js:236) alongside analytics.md and log.md, so it is gathered to the site at /develop/python/otel-bridge. Its in-repo links become site routes to match the sibling pages; the three references to it were repointed. packages/client-python/docs/index.md is the site's Python page and mirrors the PyPI README's ## CLI block line for line, but had no `rocketride otel` row and none of the OpenTelemetry paragraph the README gained. Added both. The guide's ## Limitations still said span timestamps are arrival time because the wire carries none; the previous commit made spans engine-timed, so that bullet now describes what actually remains bridge-timed (spans closed outside an event, pre-continuum engines, and all metric points). The Python blocks in the guide are also reformatted: ruff formats Markdown code blocks as of 0.16, and CI's ruff-action is unpinned, so the file failed `ruff format --check` today at either path. Repo-wide check is clean again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(changelog): record the rocketride otel bridge under [Unreleased] CHANGELOG.md [Unreleased] ### Added is the convention for user-visible additions and is kept alphabetical by scope; a new CLI command and a new optional extra qualify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(client-python): name Grafana Tempo in the OTel backend recipes Issue #1609's acceptance list says "Jaeger/Tempo", and the guide covered Tempo only implicitly as "Grafana stacks". Tempo is the OTLP receiver a Grafana user actually points the bridge at, and it needs one non-obvious step (enable distributor.receivers.otlp) plus a separate metrics backend, so it is worth naming rather than leaving a reader to infer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sdk): address CodeRabbit review round on the rebased head Nine threads on the rebased head, all in the OTel bridge. Dispositions: - docs/otel-bridge.md:151 (Sensitive Data Exposure, CWE-522) — FIXED. The Langfuse and LangSmith recipes now prompt for the keys (printf + read -rs, spelled identically in bash and zsh) instead of embedding placeholders that operators replace with live secrets in a command line, and point at a secret manager as the better source. The base64 pipeline is now `printf '%s' ... | base64 | tr -d '\r\n'`: real Langfuse keys make the joined string 85 characters, past GNU base64's 76-character wrap, and command substitution strips only trailing newlines, so the embedded break would have produced an invalid Authorization header. - cli/commands/otel.py:207 (Sensitive Data Exposure, CWE-532) — FIXED. The startup line ran `config.endpoint` straight to stdout; an endpoint can carry userinfo credentials or a signed query string. It is now always passed through `redact_endpoint()`, which drops userinfo, query and fragment while keeping scheme/host/port and the ingest path operators actually verify. - cli/main.py:466, otelbridge/config.py:177, otelbridge/setup.py:123 (Cleartext Transmission, CWE-319) — FIXED, once, in one place. `validate_transport_security()` runs before any exporter exists (in `build_providers`, and again in the CLI so the failure is exit code 2 rather than a traceback): a credential-looking header name from `--headers` or any of the three OTEL_EXPORTER_OTLP_*HEADERS variables, plus a non-loopback cleartext endpoint for either signal, aborts startup. Header VALUES are never read or echoed — only names reach the message. Loopback stays exempt for local collectors; `--insecure` / ROCKETRIDE_OTEL_ALLOW_INSECURE=1 is the explicit opt-out and warns on stderr. - otelbridge/config.py:189 (CWE-201, redirect header forwarding) — FIXED. The OTLP/HTTP exporters now get a session that never follows redirects. `requests` strips Authorization only when the redirect changes host, so a 3xx replayed x-api-key to the target and even kept Authorization on an https -> http downgrade to the same host. - otelbridge/config.py:177 (endpoint precedence) — FIXED. `from_args_env` no longer pre-reads OTEL_EXPORTER_OTLP_ENDPOINT into an explicit exporter endpoint: doing so overrode the signal-specific _TRACES_ENDPOINT / _METRICS_ENDPOINT, inverting the OTLP spec order — the same defect already fixed for the header variables in c23b312. The SDK now resolves all three, and `effective_endpoint()` mirrors its order read-only for the startup line and the security check. - otelbridge/bridge.py:207 (signal ownership) — FIXED. `loop.add_signal_handler` replaces rather than chains, and asyncio exposes no way to restore the displaced callback, so the bridge no longer touches a signal the running loop already handles; `install_signal_handlers=False` turns registration off entirely for embedders. - otelbridge/mapper.py:611 (unbounded run.pipes) — FIXED. MAX_TRACKED_RUNS did not bound pipes: a run that keeps receiving events is re-inserted as most-recently-eventful and never evicted, while every begin (or event for a never-begun pipe) adds a _PipeState that only a matching end removes, and a reconnect does not replay missed ends. MAX_TRACKED_PIPES_PER_RUN now closes the least-recently-eventful pipe as unclosed, with recency updated on pipe activity rather than inferred from insertion order. - otelbridge/setup.py:202 (tracer leak) — FIXED. A failure anywhere in the metric half left the BatchSpanProcessor worker running with no handle to stop it, because `shutdown` had not been created yet. That path now shuts the tracer provider down before re-raising. Tests: 32 added (131 in tests/test_otel_*.py, was 99), covering stdout redaction, the refusal and its opt-out, env-var and signal-specific endpoint coverage, the non-redirecting session, the tracer teardown, signal ownership and the pipe cap's LRU behaviour. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(sdk): skip the loop signal-handler ownership test on Windows asyncio's ProactorEventLoop raises NotImplementedError from add_signal_handler, so the test cannot set up its precondition there. The bridge itself already tolerates that platform (the other signal tests pass on Windows); only the test needs the platform guard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(cli): document the otel --insecure flag in the site CLI reference The `otel` command grew an `--insecure` opt-out for the cleartext-credential guard, but the docs-site "Key flags" table was never updated to match. Users hitting exit code 2 from `rocketride otel --endpoint http://tempo:4318 --headers Authorization=...` had no way to learn why from the CLI reference — the flag was only documented in the client-python OTel bridge guide. Add the missing row, naming the `ROCKETRIDE_OTEL_ALLOW_INSECURE` env var alongside the flag so the site table matches otel-bridge.md and the argparse help text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sdk): declare the otel test dependency and stop orphaning bridge providers Two review findings on the `rocketride otel` bridge. 1. The opentelemetry dependency of the bridge tests was inherited, not declared. tests/test_otel_mapper.py and tests/test_otel_setup.py opened with pytest.importorskip('opentelemetry'), and nothing in the test path installs the 'otel' extra — the packages reached the engine interpreter only as a transitive dependency of the crewai node. Dropping or repinning that node would have turned ~1450 lines of mapper/setup coverage into a silent skip with CI still green. CI never pip-installs the package (pytest runs on the engine interpreter importing the package from the source checkout via tests/conftest.py), so the extras alone cannot fix this. The deps are added to packages/server/build-requirements.txt, which server:setup-test-deps installs through depends() under the merged constraints — the one install step that path goes through — and the 'dev'/'test' extras now pull 'rocketride[otel]' so a source checkout matches. Both importorskip guards become hard imports: a missing dependency now fails collection instead of skipping. 2. build_providers() always built both provider halves and adopted its shutdown only when shutdown_fn was None, so an embedder passing its own shutdown_fn left a TracerProvider and its BatchSpanProcessor worker thread with nothing to stop them — and the half the caller had already supplied via a factory was built, wired and discarded. build_providers() takes with_tracer / with_meter (defaulting to the old behaviour) and run_bridge asks only for the half it is missing; a caller-supplied shutdown_fn is now chained in front of the providers' shutdown — caller first, providers in a finally — instead of replacing it. The CLI path passes neither argument and is unaffected. Regression tests cover chain order, the provider shutdown surviving a raising caller, and which half is built for each factory combination; test_otel_bridge.py stays runnable without the extra. The embedded-use ownership rules are documented in the bridge doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * build(client-python): install the otel test deps in the test step, not the engine Review follow-up: the previous commit declared opentelemetry-sdk and opentelemetry-exporter-otlp-proto-http in packages/server/build-requirements.txt. That file is installed by server:setup-test-deps on every engine build, so every engine install carried the OTel SDK and the OTLP exporter whether or not anyone ran the bridge — which made the summary's "zero new base dependencies" true of the package but not of the engine. Move the cost to the step that needs it. A new client-python:setup-test-deps action installs packages/client-python/tests/requirements.txt into the engine interpreter through depends(), and runs only inside client-python:test, between the parallel build group and the test-server bracket: server:build -> [nodes:build | ai:build | client-python:build] -> client-python:setup-test-deps -> py-test-server:setup -> client-python:run-pytest -> teardown That ordering puts it after the node/ai requirement files land in the dist, so depends() resolves against the complete constraint set, and before the bracket, so a cold install does not hold the test server open. This mirrors the existing packages/ai/tests/requirements.txt pattern (packages/ai/scripts/tasks.js). packages/server/build-requirements.txt is restored byte-for-byte to its previous state. The 'rocketride[otel]' self-reference in the dev/test extras stays — that is the source-checkout path — and the hard imports in test_otel_mapper.py / test_otel_setup.py stay, so a missing dependency still fails collection instead of skipping. The tests no longer depend on the crewai node pulling opentelemetry in transitively either way. Docstrings in both test modules and the extras comment in pyproject.toml now point at the new file and step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(client-python): surface the OTel bridge on the Python SDK overview Every other page under docs/public/python/ has a card in the SDK overview's "Explore the SDK" grid; the newly re-homed otel-bridge page did not, so it was reachable only from the sidebar and the CLI reference. Adds the card last in the grid, matching sidebar_position: 13. Uses LuRadar, which is not already taken by another card in this file and is present in the Lucide set react-icons v5 ships. This replaces the "OpenTelemetry export (`rocketride otel`)" section that used to live in docs/README-python-client.md and packages/client-python/docs/index.md — both deleted by the docs revamp (#2139). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(clients): declare otel-bridge as a Python-only site page The client-docs gate requires both folders to publish the same page set, with single-language extras declared in two places: PY_ONLY_PAGES/TS_ONLY_PAGES in scripts/validate-client-docs.py, and the Site page set section of the schema. docs/public/python/otel-bridge.md was added without either, so 'Docs site build' failed with: python: undeclared site page otel-bridge.md. The page is genuinely Python-only — the TypeScript SDK ships no OTel bridge (no otel surface under packages/client-typescript/src, no page in docs/public/typescript/) — so it belongs in PY_ONLY_PAGES rather than being mirrored. Declared at sidebar_position 13, matching the TypeScript-only database-sequelize extra. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(sdk): name the private SDK attribute the OTel bridge hooks The bridge receives monitor events by assigning its dispatcher to RocketRideClient._caller_on_event — the attribute EventMixin.__init__ creates from the public on_event= kwarg and EventMixin.on_event awaits for every DAP envelope. It assigns rather than passing on_event= because it is handed an already-built client and must restore the previous handler on exit. Nothing in the SDK's public surface pins that name, and a rename would fail silently: assigning an unknown attribute to a Python object succeeds, so the bridge would keep connecting, keep subscribing, keep logging "reconnected", and never export another span. Lift the string into CLIENT_EVENT_HOOK with the rationale next to it, so the coupling is a single named seam a test can assert on instead of a literal repeated across three call sites. No behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(sdk): pin the OTel bridge against the real RocketRideClient The otel suite was mock-only past the mapper: test_otel_bridge.py and test_otel_cli.py each define their own FakeClient, and the provider tests monkeypatch build_providers, FlowSpanMapper and MetricsMapper out. That covers the run loop's branches but leaves the bridge's one point of coupling to the SDK untested — and a fake that defines _caller_on_event itself can never notice the SDK dropping it. Measured: renaming _caller_on_event to _caller_on_event_renamed across mixins/events.py and mixins/data.py — a change that stops the bridge exporting anything at all — leaves all 141 existing otel tests green. Add tests/test_otel_client_contract.py, which constructs a REAL RocketRideClient and stubs only the transport (call/is_connected); the SDK's add_monitor bookkeeping, EventMixin.on_event dispatch, the mappers and the OpenTelemetry provider pipeline all run for real: - three tests pinning CLIENT_EVENT_HOOK from different angles: the attribute exists in vars() on a real client, it is where the public on_event= kwarg lands, and the SDK's dispatch actually awaits it - run_bridge end to end on a real client, with events pushed through EventMixin.on_event rather than a fake's emit(), asserting routing, the unconsumed-event case, and hook restoration on shutdown - the wildcard subscription as the real SDK encodes it on the wire, plus the _monitor_keys entry the bridge relies on for replay after a reconnect - the 24-record wire fixture replayed to real exported spans, matching the 10-span forest test_otel_mapper.py gets by feeding the mappers directly, so the bridge provably adds, drops and reorders nothing Under the same simulated rename, six of these seven fail. Also add an opt-in live export against a real OTLP collector, gated on ROCKETRIDE_OTEL_TEST_ENDPOINT via the same skipif shape RocketRideClient_test.py uses for requires_llm. It starts no collector and changes no CI step; without the env var it simply skips. Note the new file in the client-python:setup-test-deps comment (it imports the OTel SDK at collection time) and cross-reference it from test_otel_bridge.py's docstring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: nihalnihalani <24360630+nihalnihalani@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…mented ingester protocol (zero engine changes) (rocketride-org#1612) * feat(sdk): rocketride.otelbridge — OpenTelemetry export over the documented ingester protocol (rocketride-org#1609) Maps the engine's monitor event stream to standard OTel telemetry with zero engine/server changes — the client-side refinement invited by tracking issue rocketride-org#616 after the engine-invasive rocketride-org#498 was archived. - mapper.py: stateful FlowSpanMapper reconstructs span trees from apaevt_flow — task root span, pipe span per pipe id, component child spans paired by component identity (per the observability doc's warning against stack-position pairing); trace.error -> span status ERROR + exception event; apaevt_sse -> span events; unclosed spans at end/shutdown flagged rocketride.span.unclosed=true; per-pipe state is bounded. MetricsMapper maps apaevt_status_update snapshots to OTel instruments (object counts, rates, cpu/gpu memory + peaks). - Semantic conventions per the July 2026 semantic-conventions-genai snapshot: gen_ai.provider.name (not the deprecated gen_ai.system), gen_ai.operation.name from the well-known list only, span names '{operation} {model}' style, INTERNAL span kind for in-process component executions. - Privacy by default: lane payloads / prompt / result content never reach attributes or events unless include_content is set (8KB truncation cap when it is); enforced by a sentinel-scan test across all exported attributes and events. - setup.py: TracerProvider/MeterProvider + OTLP http/protobuf exporters; ALL opentelemetry imports are lazy so the base package imports without the extra. 53 unit tests (InMemorySpanExporter/InMemoryMetricReader; importorskip so the base CI matrix stays green), including a full replay of the captured real-engine event fixture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(cli): add 'rocketride otel' subcommand + optional [otel] extra (rocketride-org#1609) rocketride otel [--endpoint URL] [--protocol http|grpc] [--service-name] [--headers k=v,...] [--include-content] [--no-metrics] + the standard connection args. Subscribes via rrext_monitor with token '*' (the observability doc's recommended ingestion scope, types TASK|SUMMARY| FLOW|SSE), dispatches to the mappers, reconnects with capped backoff and resubscribes (subscriptions are per-connection and not durable), and flushes exporters on SIGINT/SIGTERM. Endpoint resolution honors the standard OTLP env contract: explicit --endpoint > OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (verbatim) > OTEL_EXPORTER_OTLP_ENDPOINT (base URL + /v1/traces|/v1/metrics) > default. Packaging: opentelemetry-sdk + OTLP http exporter live behind an optional extra — the base install gains zero dependencies, and running the subcommand without the extra exits 2 with the exact "pip install 'rocketride[otel]'" hint before any connection attempt. 25 tests: config precedence matrix, dispatch/reconnect/resubscribe with a fake client, shutdown flush ordering, missing-extra exit path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: OpenTelemetry bridge guide with Jaeger, Langfuse, and Datadog recipes Setup guide covering the span model (task -> pipe -> component tree), the privacy default and --include-content, the FLOW-requires-trace-level callout, reconnect semantics, and copy-paste recipes with exact OTLP endpoints/auth for Jaeger all-in-one, Langfuse (/api/public/otel, Basic auth), and Datadog. Notes that Jaeger does not ingest OTLP metrics — pair it with --no-metrics or point metrics at a collector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): handle partial OpenTelemetry installs — guard the HTTP exporter import CI exposed a real environment our matrix missed: opentelemetry-api/sdk present as another package's transitive dependency (tool_daytona pulls them in) while the OTLP HTTP exporter is absent. Two fixes: - setup.py: the http trace/metric exporter imports are now guarded like the grpc ones — a partial install raises OtelNotInstalledError with the pip install 'rocketride[otel]' hint (CLI exit 2) instead of a raw ModuleNotFoundError. Verified live in a reproduced partial venv. - test_otel_setup.py: importorskip targets the most specific module the tests exercise (opentelemetry.exporter.otlp.proto.http), so the module skips cleanly in CI's partial environment; plus a regression test that simulates the partial install via a None sys.modules entry and asserts the friendly error. Partial venv: 63 passed + 1 module skip; full extra: 79 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): OTel bridge review round — privacy-gate errors, run-id promotion, header precedence, shutdown ordering CodeRabbit review round on rocketride-org#1612, each finding verified before acting: - mapper: trace.error no longer bypasses the privacy gate — by default only the error signal exports (status ERROR, bare exception event, generic description); --include-content restores verbatim text, now subject to the same 8KB truncation strings previously skipped. The sentinel privacy test plants the sentinel in trace.error. - mapper: a run first seen via flow events (before the task 'running' snapshot) is now promoted from its project|source fallback key to the canonical run id when it arrives — previously a second run state was created (broken parentage) and the orphaned fallback state leaked permanently. Confirmed with a failing repro; guard test ensures two distinct canonical runs sharing (project, source) never merge. - config/setup: only --headers become explicit exporter headers; env headers are left to the SDK so OTEL_EXPORTER_OTLP_TRACES_HEADERS / _METRICS_HEADERS signal precedence works (it was silently defeated). Precedence matrix tests added. - setup: meter provider shutdown wrapped in finally so metrics flush even when tracer shutdown raises. - CLI: --trace-level none message now says flow tracing stays disabled; docs recipes pass collector auth via OTEL_EXPORTER_OTLP_HEADERS env instead of argv (shell-history hygiene); error-mapping doc wording matches the implementation (component leave events). Full extra: 90 passed. Partial install (api/sdk without exporter): 69 passed + clean skip. ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): absorb stranded fallback run state when canonical state precedes it Follow-on to the run-promotion fix: an SSE event creates the canonical run state WITHOUT registering the (project, source) alias (SSE bodies carry no project/source per the protocol), so an id-less flow for the same run could still open a parallel fallback-keyed state — and because the canonical state already existed, the promotion branch was skipped, stranding the fallback state (kept alive forever by snapshot reconciliation) with duplicate spans. _get_run now reconciles whenever an event carries both the canonical id and project/source: no canonical state → promote (unchanged); canonical state exists → _absorb_fallback_run migrates non-colliding pipes (open spans re-stamped with the canonical run id), closes colliding pipes as rocketride.span.unclosed=true, and drops the fallback key. Only fallback-format keys are touched, so distinct canonical runs never merge (guard test unchanged and green). Both new regression tests failed before the fix. Full extra: 92 passed; partial install: 71 passed + expected skip; ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): stamp OTel spans with the engine's body.eventTime The bridge was written against a monitor protocol that carried no event timestamps, so every span used the bridge's own clock at event arrival and the guide documented that as a limitation. Since rocketride-org#1661 the engine stamps `body.eventTime` (epoch seconds, set once at ingress by the run-log continuum) on every forwarded event, and derived flow events inherit their source event's stamp — so the limitation is no longer true and the documentation shipped with it was wrong against develop. Span start/end and span-event timestamps now come from `body.eventTime`, making durations engine-measured instead of bridge-observed: WebSocket transport latency and event-loop scheduling no longer inflate every component span. The fallback is exactly the previous behaviour. A body with no usable `eventTime` (an older engine, or a non-float/NaN/non-positive value) and every span closed outside an event — shutdown, seeded-snapshot reconciliation, tracked-run cap eviction — use `time.time_ns()`, the same clock the SDK would have used. Because a run can therefore mix both sources, each open span records its own start and a close is clamped to never precede it, so no span can report a negative duration. Metric points are unchanged: OpenTelemetry metric instruments take no explicit timestamp, so they remain arrival-timed. That is now stated rather than left implicit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(client-python): publish the OTel bridge guide and mirror its CLI entry Two co-location gaps (AGENTS.md:5-28) and one stale claim. The 369-line guide lived at docs/README-otel-bridge.md, and root docs/ is not in the docs-site glob (packages/docs/scripts/lib/gather.js:25), so the guide was GitHub-only. Moved to packages/client-python/docs/otel-bridge.md, the tree already mounted at develop/python (packages/client-python/scripts/ tasks.js:236) alongside analytics.md and log.md, so it is gathered to the site at /develop/python/otel-bridge. Its in-repo links become site routes to match the sibling pages; the three references to it were repointed. packages/client-python/docs/index.md is the site's Python page and mirrors the PyPI README's ## CLI block line for line, but had no `rocketride otel` row and none of the OpenTelemetry paragraph the README gained. Added both. The guide's ## Limitations still said span timestamps are arrival time because the wire carries none; the previous commit made spans engine-timed, so that bullet now describes what actually remains bridge-timed (spans closed outside an event, pre-continuum engines, and all metric points). The Python blocks in the guide are also reformatted: ruff formats Markdown code blocks as of 0.16, and CI's ruff-action is unpinned, so the file failed `ruff format --check` today at either path. Repo-wide check is clean again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(changelog): record the rocketride otel bridge under [Unreleased] CHANGELOG.md [Unreleased] ### Added is the convention for user-visible additions and is kept alphabetical by scope; a new CLI command and a new optional extra qualify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(client-python): name Grafana Tempo in the OTel backend recipes Issue rocketride-org#1609's acceptance list says "Jaeger/Tempo", and the guide covered Tempo only implicitly as "Grafana stacks". Tempo is the OTLP receiver a Grafana user actually points the bridge at, and it needs one non-obvious step (enable distributor.receivers.otlp) plus a separate metrics backend, so it is worth naming rather than leaving a reader to infer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sdk): address CodeRabbit review round on the rebased head Nine threads on the rebased head, all in the OTel bridge. Dispositions: - docs/otel-bridge.md:151 (Sensitive Data Exposure, CWE-522) — FIXED. The Langfuse and LangSmith recipes now prompt for the keys (printf + read -rs, spelled identically in bash and zsh) instead of embedding placeholders that operators replace with live secrets in a command line, and point at a secret manager as the better source. The base64 pipeline is now `printf '%s' ... | base64 | tr -d '\r\n'`: real Langfuse keys make the joined string 85 characters, past GNU base64's 76-character wrap, and command substitution strips only trailing newlines, so the embedded break would have produced an invalid Authorization header. - cli/commands/otel.py:207 (Sensitive Data Exposure, CWE-532) — FIXED. The startup line ran `config.endpoint` straight to stdout; an endpoint can carry userinfo credentials or a signed query string. It is now always passed through `redact_endpoint()`, which drops userinfo, query and fragment while keeping scheme/host/port and the ingest path operators actually verify. - cli/main.py:466, otelbridge/config.py:177, otelbridge/setup.py:123 (Cleartext Transmission, CWE-319) — FIXED, once, in one place. `validate_transport_security()` runs before any exporter exists (in `build_providers`, and again in the CLI so the failure is exit code 2 rather than a traceback): a credential-looking header name from `--headers` or any of the three OTEL_EXPORTER_OTLP_*HEADERS variables, plus a non-loopback cleartext endpoint for either signal, aborts startup. Header VALUES are never read or echoed — only names reach the message. Loopback stays exempt for local collectors; `--insecure` / ROCKETRIDE_OTEL_ALLOW_INSECURE=1 is the explicit opt-out and warns on stderr. - otelbridge/config.py:189 (CWE-201, redirect header forwarding) — FIXED. The OTLP/HTTP exporters now get a session that never follows redirects. `requests` strips Authorization only when the redirect changes host, so a 3xx replayed x-api-key to the target and even kept Authorization on an https -> http downgrade to the same host. - otelbridge/config.py:177 (endpoint precedence) — FIXED. `from_args_env` no longer pre-reads OTEL_EXPORTER_OTLP_ENDPOINT into an explicit exporter endpoint: doing so overrode the signal-specific _TRACES_ENDPOINT / _METRICS_ENDPOINT, inverting the OTLP spec order — the same defect already fixed for the header variables in c23b312. The SDK now resolves all three, and `effective_endpoint()` mirrors its order read-only for the startup line and the security check. - otelbridge/bridge.py:207 (signal ownership) — FIXED. `loop.add_signal_handler` replaces rather than chains, and asyncio exposes no way to restore the displaced callback, so the bridge no longer touches a signal the running loop already handles; `install_signal_handlers=False` turns registration off entirely for embedders. - otelbridge/mapper.py:611 (unbounded run.pipes) — FIXED. MAX_TRACKED_RUNS did not bound pipes: a run that keeps receiving events is re-inserted as most-recently-eventful and never evicted, while every begin (or event for a never-begun pipe) adds a _PipeState that only a matching end removes, and a reconnect does not replay missed ends. MAX_TRACKED_PIPES_PER_RUN now closes the least-recently-eventful pipe as unclosed, with recency updated on pipe activity rather than inferred from insertion order. - otelbridge/setup.py:202 (tracer leak) — FIXED. A failure anywhere in the metric half left the BatchSpanProcessor worker running with no handle to stop it, because `shutdown` had not been created yet. That path now shuts the tracer provider down before re-raising. Tests: 32 added (131 in tests/test_otel_*.py, was 99), covering stdout redaction, the refusal and its opt-out, env-var and signal-specific endpoint coverage, the non-redirecting session, the tracer teardown, signal ownership and the pipe cap's LRU behaviour. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(sdk): skip the loop signal-handler ownership test on Windows asyncio's ProactorEventLoop raises NotImplementedError from add_signal_handler, so the test cannot set up its precondition there. The bridge itself already tolerates that platform (the other signal tests pass on Windows); only the test needs the platform guard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(cli): document the otel --insecure flag in the site CLI reference The `otel` command grew an `--insecure` opt-out for the cleartext-credential guard, but the docs-site "Key flags" table was never updated to match. Users hitting exit code 2 from `rocketride otel --endpoint http://tempo:4318 --headers Authorization=...` had no way to learn why from the CLI reference — the flag was only documented in the client-python OTel bridge guide. Add the missing row, naming the `ROCKETRIDE_OTEL_ALLOW_INSECURE` env var alongside the flag so the site table matches otel-bridge.md and the argparse help text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sdk): declare the otel test dependency and stop orphaning bridge providers Two review findings on the `rocketride otel` bridge. 1. The opentelemetry dependency of the bridge tests was inherited, not declared. tests/test_otel_mapper.py and tests/test_otel_setup.py opened with pytest.importorskip('opentelemetry'), and nothing in the test path installs the 'otel' extra — the packages reached the engine interpreter only as a transitive dependency of the crewai node. Dropping or repinning that node would have turned ~1450 lines of mapper/setup coverage into a silent skip with CI still green. CI never pip-installs the package (pytest runs on the engine interpreter importing the package from the source checkout via tests/conftest.py), so the extras alone cannot fix this. The deps are added to packages/server/build-requirements.txt, which server:setup-test-deps installs through depends() under the merged constraints — the one install step that path goes through — and the 'dev'/'test' extras now pull 'rocketride[otel]' so a source checkout matches. Both importorskip guards become hard imports: a missing dependency now fails collection instead of skipping. 2. build_providers() always built both provider halves and adopted its shutdown only when shutdown_fn was None, so an embedder passing its own shutdown_fn left a TracerProvider and its BatchSpanProcessor worker thread with nothing to stop them — and the half the caller had already supplied via a factory was built, wired and discarded. build_providers() takes with_tracer / with_meter (defaulting to the old behaviour) and run_bridge asks only for the half it is missing; a caller-supplied shutdown_fn is now chained in front of the providers' shutdown — caller first, providers in a finally — instead of replacing it. The CLI path passes neither argument and is unaffected. Regression tests cover chain order, the provider shutdown surviving a raising caller, and which half is built for each factory combination; test_otel_bridge.py stays runnable without the extra. The embedded-use ownership rules are documented in the bridge doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * build(client-python): install the otel test deps in the test step, not the engine Review follow-up: the previous commit declared opentelemetry-sdk and opentelemetry-exporter-otlp-proto-http in packages/server/build-requirements.txt. That file is installed by server:setup-test-deps on every engine build, so every engine install carried the OTel SDK and the OTLP exporter whether or not anyone ran the bridge — which made the summary's "zero new base dependencies" true of the package but not of the engine. Move the cost to the step that needs it. A new client-python:setup-test-deps action installs packages/client-python/tests/requirements.txt into the engine interpreter through depends(), and runs only inside client-python:test, between the parallel build group and the test-server bracket: server:build -> [nodes:build | ai:build | client-python:build] -> client-python:setup-test-deps -> py-test-server:setup -> client-python:run-pytest -> teardown That ordering puts it after the node/ai requirement files land in the dist, so depends() resolves against the complete constraint set, and before the bracket, so a cold install does not hold the test server open. This mirrors the existing packages/ai/tests/requirements.txt pattern (packages/ai/scripts/tasks.js). packages/server/build-requirements.txt is restored byte-for-byte to its previous state. The 'rocketride[otel]' self-reference in the dev/test extras stays — that is the source-checkout path — and the hard imports in test_otel_mapper.py / test_otel_setup.py stay, so a missing dependency still fails collection instead of skipping. The tests no longer depend on the crewai node pulling opentelemetry in transitively either way. Docstrings in both test modules and the extras comment in pyproject.toml now point at the new file and step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(client-python): surface the OTel bridge on the Python SDK overview Every other page under docs/public/python/ has a card in the SDK overview's "Explore the SDK" grid; the newly re-homed otel-bridge page did not, so it was reachable only from the sidebar and the CLI reference. Adds the card last in the grid, matching sidebar_position: 13. Uses LuRadar, which is not already taken by another card in this file and is present in the Lucide set react-icons v5 ships. This replaces the "OpenTelemetry export (`rocketride otel`)" section that used to live in docs/README-python-client.md and packages/client-python/docs/index.md — both deleted by the docs revamp (rocketride-org#2139). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(clients): declare otel-bridge as a Python-only site page The client-docs gate requires both folders to publish the same page set, with single-language extras declared in two places: PY_ONLY_PAGES/TS_ONLY_PAGES in scripts/validate-client-docs.py, and the Site page set section of the schema. docs/public/python/otel-bridge.md was added without either, so 'Docs site build' failed with: python: undeclared site page otel-bridge.md. The page is genuinely Python-only — the TypeScript SDK ships no OTel bridge (no otel surface under packages/client-typescript/src, no page in docs/public/typescript/) — so it belongs in PY_ONLY_PAGES rather than being mirrored. Declared at sidebar_position 13, matching the TypeScript-only database-sequelize extra. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(sdk): name the private SDK attribute the OTel bridge hooks The bridge receives monitor events by assigning its dispatcher to RocketRideClient._caller_on_event — the attribute EventMixin.__init__ creates from the public on_event= kwarg and EventMixin.on_event awaits for every DAP envelope. It assigns rather than passing on_event= because it is handed an already-built client and must restore the previous handler on exit. Nothing in the SDK's public surface pins that name, and a rename would fail silently: assigning an unknown attribute to a Python object succeeds, so the bridge would keep connecting, keep subscribing, keep logging "reconnected", and never export another span. Lift the string into CLIENT_EVENT_HOOK with the rationale next to it, so the coupling is a single named seam a test can assert on instead of a literal repeated across three call sites. No behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(sdk): pin the OTel bridge against the real RocketRideClient The otel suite was mock-only past the mapper: test_otel_bridge.py and test_otel_cli.py each define their own FakeClient, and the provider tests monkeypatch build_providers, FlowSpanMapper and MetricsMapper out. That covers the run loop's branches but leaves the bridge's one point of coupling to the SDK untested — and a fake that defines _caller_on_event itself can never notice the SDK dropping it. Measured: renaming _caller_on_event to _caller_on_event_renamed across mixins/events.py and mixins/data.py — a change that stops the bridge exporting anything at all — leaves all 141 existing otel tests green. Add tests/test_otel_client_contract.py, which constructs a REAL RocketRideClient and stubs only the transport (call/is_connected); the SDK's add_monitor bookkeeping, EventMixin.on_event dispatch, the mappers and the OpenTelemetry provider pipeline all run for real: - three tests pinning CLIENT_EVENT_HOOK from different angles: the attribute exists in vars() on a real client, it is where the public on_event= kwarg lands, and the SDK's dispatch actually awaits it - run_bridge end to end on a real client, with events pushed through EventMixin.on_event rather than a fake's emit(), asserting routing, the unconsumed-event case, and hook restoration on shutdown - the wildcard subscription as the real SDK encodes it on the wire, plus the _monitor_keys entry the bridge relies on for replay after a reconnect - the 24-record wire fixture replayed to real exported spans, matching the 10-span forest test_otel_mapper.py gets by feeding the mappers directly, so the bridge provably adds, drops and reorders nothing Under the same simulated rename, six of these seven fail. Also add an opt-in live export against a real OTLP collector, gated on ROCKETRIDE_OTEL_TEST_ENDPOINT via the same skipif shape RocketRideClient_test.py uses for requires_llm. It starts no collector and changes no CI step; without the env var it simply skips. Note the new file in the client-python:setup-test-deps comment (it imports the OTel SDK at collection time) and cross-reference it from test_otel_bridge.py's docstring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: nihalnihalani <24360630+nihalnihalani@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#Hack-with-bay-2
Contribution Type
Feature — Add OpenTelemetry distributed tracing and Prometheus metrics export
Problem / Use Case
RocketRide Server has an internal
MetricsManager(packages/ai/src/ai/web/metrics/metrics.py) that tracks per-task timers and counters, but this data is only accessible within the engine. There is no way to:Proposed Solution
prometheus_metrics.py) — 9 metrics covering pipelines, LLM requests/tokens, vector DB ops, HTTP requests. Custom histogram buckets for LLM durations (0.1s to 120s).tracing.py) — Configurable OTLP/console/none exporter with thread-safe setup viathreading.Lock. Auto-instruments FastAPI.middleware.py) — Automatic HTTP request instrumentation with URL path normalization to prevent label cardinality explosion (UUIDs and numeric IDs replaced with/{id})./metricsendpoint (metrics_endpoint.py) — Prometheus text exposition format, configurable public/private access with security documentation.Value Added
rocketride_llm_tokens_totalby provider/model enables per-model cost analysisOTEL_EXPORTER_TYPE=none(default) adds no overheadWhy This Feature Fits This Codebase
The existing
MetricsManageratpackages/ai/src/ai/web/metrics/metrics.pyalready tracks timers and counters per-task/per-pipe (lines 96-423). This PR extends that foundation with industry-standard export formats without modifying the existing class.The FastAPI server at
packages/ai/src/ai/web/server.pyalready uses middleware (AuthMiddlewareat line 20). The newMetricsMiddlewarefollows the identical StarletteBaseHTTPMiddlewarepattern. URL path normalization uses Starlette's route matching (route.matches(scope)) to resolve paths to templates, preventing the cardinality explosion that would occur with dynamic path parameters like/pipeline/{id}/status.The
/metricsendpoint follows the same registration pattern as existing endpoints inpackages/ai/src/ai/web/endpoints/(ping, version, status, shutdown).Validation
Tests verify: counter increments, histogram observations, gauge operations, Prometheus text format output, all 3 exporter types, middleware request recording for 200/404/500 status codes.
How This Could Be Extended
rocketride_circuit_breaker_stategaugedeploy/grafana/🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Metrics
Observability
Tests