Skip to content

fix(engine): run node teardown when pipelines are cancelled - #2396

Merged
Leela8256 merged 1 commit into
developfrom
fix/RR-2300-pipeline-cancel-teardown
Sep 22, 2026
Merged

Leela8256 merged 1 commit into
developfrom
fix/RR-2300-pipeline-cancel-teardown

Conversation

@Leela8256

@Leela8256 Leela8256 commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Type

Bug fix. Adds rocketlib.isCancelled() for long-running Python sources. The existing five-second graceful-stop deadline and forced-kill fallback remain in place.

Testing

  • Tests added or updated
  • Tested locally
  • ./builder test passes — the full monorepo suite was not run; targeted checks below passed.

Validation on macOS arm64:

  • Built the engine from source and compiled the native test executable.
  • Native regression tests: 27 assertions passed in 3 test cases, covering teardown on success/error/cancellation, error precedence, and the Python cancellation binding.
  • Python tests using the built engine: 48 passed across the webhook and rocketlib media-normalization suites; another 24 README-validator tests passed.
  • Live, isolated EaaS probe: Chat and Webhook each recorded exactly one beginGlobal() and one endGlobal() on both explicit termination and idle-TTL expiry, without sending requests or using vendor resources. Explicit termination completed in 0.289 s and 0.369 s; TTL cleanup ran at the normal 60-second monitor cycle. All four task processes exited.
  • Ruff lint/format, webhook README validation, git diff --check, ./builder docs:check, and ./builder docs:build passed.
  • Local CodeRabbit review against develop: addressed the initial test-docstring finding; the second full review completed with zero findings.

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included — pre-commit gitleaks passed
  • Wiki updated (if applicable) — lifecycle documentation and webhook README updated
  • Breaking changes documented (if applicable) — no breaking API changes

Linked Issue

Fixes #2300

Summary by CodeRabbit

  • New Features

    • Added cancellation status checks for long-running Python sources, allowing them to stop promptly when execution is cancelled.
    • Webhook processing now responds to cancellation during shutdown and reliably performs cleanup.
  • Bug Fixes

    • Task cleanup now runs after failed or cancelled executions.
    • Execution errors take precedence over cleanup errors when both occur.
    • Cancellation is correctly reported even when a source produces no objects.
  • Documentation

    • Expanded guidance on cancellation handling, cleanup deadlines, and external resource timeouts.

@github-actions github-actions Bot added docs Documentation module:server C++ engine and server components module:nodes Python pipeline nodes labels Sep 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: rocketride-org/rocketride-server/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ff1e94de-348f-4d3d-a433-cf3fae437e2d

📥 Commits

Reviewing files that changed from the base of the PR and between 274402f and 2784ca9.

📒 Files selected for processing (12)
  • docs/development/nodes/index.md
  • nodes/src/nodes/webhook/IEndpoint.py
  • nodes/src/nodes/webhook/README.md
  • nodes/test/webhook/test_cancellation.py
  • packages/server/engine-lib/engLib/store/python/bindings.cpp
  • packages/server/engine-lib/engLib/store/python/python-endpoint.source.cpp
  • packages/server/engine-lib/engLib/task/core/task.cpp
  • packages/server/engine-lib/rocketlib-python/lib/rocketlib/__init__.py
  • packages/server/engine-lib/rocketlib-python/lib/rocketlib/__init__.pyi
  • packages/server/engine-lib/rocketlib-python/lib/rocketlib/engine.py
  • packages/server/engine-lib/test/python/engine.cpp
  • packages/server/engine-lib/test/task/lifecycle.cpp

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

The change exposes native cancellation to Python, propagates cancellation from Python sources, ensures task teardown after execution failures, and updates the webhook source and documentation to support cancellation-aware cleanup.

Changes

Cancellation-aware teardown

Layer / File(s) Summary
Cancellation status contract
packages/server/engine-lib/engLib/store/python/bindings.cpp, packages/server/engine-lib/rocketlib-python/lib/rocketlib/..., packages/server/engine-lib/engLib/store/python/python-endpoint.source.cpp, packages/server/engine-lib/test/python/engine.cpp
The engine exposes rocketlib.isCancelled(). Python endpoint scans now report cancellation even when no object reaches the callback. Tests verify the public helper and native cancellation state.
Task teardown on execution failure
packages/server/engine-lib/engLib/task/core/task.cpp, packages/server/engine-lib/test/task/lifecycle.cpp
ITask::execute() calls endTask() after exec() errors or cancellation. The execution error takes precedence when both phases fail. Lifecycle tests cover begin, execution, and teardown outcomes.
Webhook cancellation and lifecycle documentation
nodes/src/nodes/webhook/IEndpoint.py, nodes/test/webhook/test_cancellation.py, docs/development/nodes/index.md, nodes/src/nodes/webhook/README.md
The webhook source polls cancellation and runs _shutdown() in finally. Tests cover cancellation during waits, pre-existing cancellation, and wait failures. Documentation describes teardown and graceful-stop deadlines.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 2784c

Chat and Webhook cancellations now reach cleanup, and failed executions still run task teardown; no merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 72.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 10 files. (2 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: ensuring node teardown runs when pipelines are cancelled.
Linked Issues check ✅ Passed Issue #2300 requires cancellation-aware Chat/Webhook sources, teardown after cancellation or execution failure, execution-error precedence, preserved graceful-stop behavior, lifecycle documentation, a…
Out of Scope Changes check ✅ Passed The changed files support Issue #2300. They implement cancellation observation, task teardown, the Python cancellation API, regression tests, and lifecycle documentation. No unrelated caller-identity …
Full details: Docstring Coverage

Explanation

Docstring coverage is 72.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 10 files. (2 skipped: 2 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Leela8256
Leela8256 merged commit da6192d into develop Sep 22, 2026
26 checks passed
@Leela8256
Leela8256 deleted the fix/RR-2300-pipeline-cancel-teardown branch September 22, 2026 16:29
2001anshkaushik added a commit that referenced this pull request Oct 20, 2026
Tenki's idle timeout never paused anything. Its changelog of 2026-09-18
removed the --idle-timeout option as having "never had any effect", and
SDK 1.4.0 warns that idle_timeout_minutes is deprecated and ignored and
no longer sends it; 1.0.6, the pinned version, sent it and the service
ignored it. The node's idle_timeout_minutes field, its "pauses the
session, which stops compute billing" wording, and the README's cost
guidance were therefore wrong: an unused session bills until it is
closed or reaches max_duration.

The field is removed from the schema, the create call and the docs. A
pipeline saved with it still starts: the key is simply not read, which
a regression test pins. The recovery path for a paused session stays,
since a session can still be paused by hand or by Tenki when a
workspace runs out of credit; only its docstring changes.

max_duration_minutes now defaults to 15 instead of 60. With no idle
pause, it is the only Tenki-side bound on what a session the pipeline
never closed can bill: after an engine crash on any engine, and on
terminate and idle-TTL expiry for chat- and webhook-sourced pipelines on
engines before #2396 (merged to develop on 2026-09-22; this branch's
base predates it). The README and IGlobal docstring now say exactly
that, instead of "teardown never runs". Validation bounds (1 to 1440)
are unchanged, and the example pipeline keeps its explicit 30.

The SDK pin stays at 1.0.6. Not touched: the generated README block and
CHANGELOG.md.
2001anshkaushik added a commit that referenced this pull request Oct 20, 2026
…use retention

Tenki pauses a session at max_duration rather than ending it (measured),
and since #2396 the engine runs teardown for chat and webhook pipelines
too. A 15 minute default no longer bought much cost safety, and every
pause cuts a running command off and costs about a minute to resume.
The default goes back to 60 minutes.

A command allowed to run as long as the session lives can never finish,
so the node now warns at startup, and in the editor's config check,
when exec_timeout_secs is not shorter than max_duration.

New setting pause_retention_hours, default 24, sent as pause_retention:
without it Tenki keeps a paused session's state for seven days. Tenki
refuses 0 and accepted every value tried from one minute to a year
(measured 2026-10-09), so the field takes 1 to 8760 hours.
2001anshkaushik added a commit that referenced this pull request Oct 20, 2026
The README said engines with #2396 run node teardown for every source.
#2396 runs teardown when a pipeline is cancelled, and its own notes say
a long-running Python source must return from scanObjects on
cancellation for that to happen (chat, webhook, dropper and tools do).
Teardown was measured here for chat- and webhook-sourced pipelines, on
terminate and on idle-TTL expiry, so the sentence now says that.
AviNormie pushed a commit to AviNormie/rocketride-server that referenced this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation module:nodes Python pipeline nodes module:server C++ engine and server components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Node endGlobal never runs for chat/webhook-sourced pipelines, so nodes cannot release external resources

2 participants