Skip to content

fix(engine): keep logs on box and K8s cancels - #897

Open
paperclover wants to merge 4 commits into
mainfrom
clo/run-engine-followups
Open

paperclover wants to merge 4 commits into
mainfrom
clo/run-engine-followups

Conversation

@paperclover

@paperclover paperclover commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes from real-provider runs after #862, all with the engine on. A cancelled Kubernetes run now keeps its log instead of sitting on "Saving the log from the host…", and no longer raises a "Deleted Job still has pods" notice. OpenResearch box runs publish their kept log before the rental is released, so the dashboard serves it afterwards instead of a 503, and they stay Starting until the box reports a live payload, so started_at is the payload's real start. Also keeps the "Go to start" tooltip inside narrow log panes.

The bugs came from smoke runs on k3s (EC2) and an OpenResearch CPU box; the Kubernetes fix was rechecked on k3s, the box fixes only against fakes so far. Full Rust suite passes with the setting off and with ORX_RUN_ENGINE=1.

Assisted-by: gpt-6.1-sol

…ng a Kubernetes run

Foreground deletion keeps the Job visible while Kubernetes removes its Pods, so normal cancellation does not look like an unexpectedly deleted Job. Once both are gone, finalize the log already collected in the local spool instead of waiting forever for a Pod that cannot return.

Finish the test’s in-flight log drain before deleting its fake Pod; Linux may still be starting kubectl after the observation’s short drain window.

Assisted-by: gpt-6.1-sol
Publish mirrored bytes as kept ranges and settle completed collection before acknowledging the remote supervisor. Cancelled boxes retain their last mirrored log without retrying the deleted host.

Assisted-by: gpt-6.1-sol
An accepted bootstrap only starts the remote supervisor. Keep pending SSH jobs in Starting until inspection finds a live payload PID, so engine start times reflect payload evidence.

Assisted-by: gpt-6.1-sol
Align the first log control’s tooltip to its start edge. Centering it puts its left edge outside a 400 px pane.

Assisted-by: gpt-6.1-sol
@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes affect run status and log retention for several remote execution paths.

Fix box cancellation without a saved EOF marker before merging; those runs can remain stuck saving their logs.

Findings

  1. P1 Cancelled logs stay pending forever ▶
Fix with agent prompt
### Issue 1
src/engine/runtime.rs:695-700
If the user cancels before the first log poll, or while the local copy still trails the remote log, `OpenResearch::cancel` stops the log-copy task and releases the box without finishing collection. This branch then only checks the saved ranges for `eof: true`. That marker can no longer arrive, so every retry returns `false` and the dashboard stays on “Saving the log from the host…” forever.

After a verified release, settle the available copy, including an empty log, instead of waiting for an EOF marker from the released box.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR keeps box logs available after release and lets cancelled Kubernetes runs keep their local log copy.

  • Cancelled Kubernetes runs keep their logs after the Job and Pods disappear.
  • OpenResearch logs stay available after a box is released.
  • Remote runs stay Starting until the payload is running.
  • The “Go to start” tooltip stays inside narrow log panes.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Cancel box run] --> B[Stop log-copy task]
  B --> C[Release rental]
  C --> D{Saved EOF marker?}
  D -->|Yes| E[Finish log collection]
  D -->|No| F[Leave collection pending]
  F --> G[Check saved ranges again]
  G --> F
Loading

Reviews (1) · Last reviewed commit: "fix(ui): keep the log-start tooltip insi..." · Reviewed by Greptile

Comment thread src/engine/runtime.rs
Comment on lines +695 to +700
if mirrored {
return Ok(
crate::commands::up::api_v2::kept_ranges(&self.data_dir, &run.id)?
.iter()
.any(|range| range.eof),
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Cancelled logs stay pending forever

If the user cancels before the first log poll, or while the local copy still trails the remote log, OpenResearch::cancel stops the log-copy task and releases the box without finishing collection. This branch then only checks the saved ranges for eof: true. That marker can no longer arrive, so every retry returns false and the dashboard stays on “Saving the log from the host…” forever.

After a verified release, settle the available copy, including an empty log, instead of waiting for an EOF marker from the released box.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/engine/runtime.rs
Line: 695-700

Comment:
**Cancelled logs stay pending forever**

If the user cancels before the first log poll, or while the local copy still trails the remote log, `OpenResearch::cancel` stops the log-copy task and releases the box without finishing collection. This branch then only checks the saved ranges for `eof: true`. That marker can no longer arrive, so every retry returns `false` and the dashboard stays on “Saving the log from the host…” forever.

After a verified release, settle the available copy, including an empty log, instead of waiting for an EOF marker from the released box.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant