Repository navigation
fix(engine): keep logs on box and K8s cancels - #897
paperclover wants to merge 4 commits into
Conversation
…ng a Kubernetes run Foreground deletion keeps the Job visible while Kubernetes removes its Pods, so normal cancellation does not look like an unexpectedly deleted Job. Once both are gone, finalize the log already collected in the local spool instead of waiting forever for a Pod that cannot return. Finish the test’s in-flight log drain before deleting its fake Pod; Linux may still be starting kubectl after the observation’s short drain window. Assisted-by: gpt-6.1-sol
Publish mirrored bytes as kept ranges and settle completed collection before acknowledging the remote supervisor. Cancelled boxes retain their last mirrored log without retrying the deleted host. Assisted-by: gpt-6.1-sol
An accepted bootstrap only starts the remote supervisor. Keep pending SSH jobs in Starting until inspection finds a live payload PID, so engine start times reflect payload evidence. Assisted-by: gpt-6.1-sol
Align the first log control’s tooltip to its start edge. Centering it puts its left edge outside a 400 px pane. Assisted-by: gpt-6.1-sol
[Medium impact] The changes affect run status and log retention for several remote execution paths. Fix box cancellation without a saved EOF marker before merging; those runs can remain stuck saving their logs. Findings
Fix with agent prompt### Issue 1
src/engine/runtime.rs:695-700
If the user cancels before the first log poll, or while the local copy still trails the remote log, `OpenResearch::cancel` stops the log-copy task and releases the box without finishing collection. This branch then only checks the saved ranges for `eof: true`. That marker can no longer arrive, so every retry returns `false` and the dashboard stays on “Saving the log from the host…” forever.
After a verified release, settle the available copy, including an empty log, instead of waiting for an EOF marker from the released box.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryThis PR keeps box logs available after release and lets cancelled Kubernetes runs keep their local log copy.
|
| if mirrored { | ||
| return Ok( | ||
| crate::commands::up::api_v2::kept_ranges(&self.data_dir, &run.id)? | ||
| .iter() | ||
| .any(|range| range.eof), | ||
| ); |
There was a problem hiding this comment.
Cancelled logs stay pending forever
If the user cancels before the first log poll, or while the local copy still trails the remote log, OpenResearch::cancel stops the log-copy task and releases the box without finishing collection. This branch then only checks the saved ranges for eof: true. That marker can no longer arrive, so every retry returns false and the dashboard stays on “Saving the log from the host…” forever.
After a verified release, settle the available copy, including an empty log, instead of waiting for an EOF marker from the released box.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/engine/runtime.rs
Line: 695-700
Comment:
**Cancelled logs stay pending forever**
If the user cancels before the first log poll, or while the local copy still trails the remote log, `OpenResearch::cancel` stops the log-copy task and releases the box without finishing collection. This branch then only checks the saved ranges for `eof: true`. That marker can no longer arrive, so every retry returns `false` and the dashboard stays on “Saving the log from the host…” forever.
After a verified release, settle the available copy, including an empty log, instead of waiting for an EOF marker from the released box.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Fixes from real-provider runs after #862, all with the engine on. A cancelled Kubernetes run now keeps its log instead of sitting on "Saving the log from the host…", and no longer raises a "Deleted Job still has pods" notice. OpenResearch box runs publish their kept log before the rental is released, so the dashboard serves it afterwards instead of a 503, and they stay Starting until the box reports a live payload, so
started_atis the payload's real start. Also keeps the "Go to start" tooltip inside narrow log panes.The bugs came from smoke runs on k3s (EC2) and an OpenResearch CPU box; the Kubernetes fix was rechecked on k3s, the box fixes only against fakes so far. Full Rust suite passes with the setting off and with
ORX_RUN_ENGINE=1.Assisted-by: gpt-6.1-sol