Skip to content

ir: Add run state fields for ephemeral runs and reused outputs - #2

Open
SCcagg5 wants to merge 8 commits into
stack/01-g1from
stack/02-g2
Open

ir: Add run state fields for ephemeral runs and reused outputs#2
SCcagg5 wants to merge 8 commits into
stack/01-g1from
stack/02-g2

Conversation

@SCcagg5

@SCcagg5 SCcagg5 commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Part of a stack of 31 pull requests that depend on each other. They must be reviewed and merged in order: each PR builds on the previous one and its diff only contains its own changes.

All PRs of the stack
# PR
1 specs: Add spec 083 and the samples store
2 ir: Add run state fields for ephemeral runs and reused outputs
3 runtime: Bound continuous steps and carry outputs of selected steps
4 api: Add step samples, events, availability and selected-step runs
5 test(ui): add YAML fixtures for the document model tests
6 ui(lib): add shared UI kit, status helpers and right-panel shell
7 ui(canvas): add canvas engine core, toolbars and editor helpers
8 ui(i18n): add workspace message catalogs, first batch
9 ui(canvas): add canvas Flow view, step run panel and workflow panel
10 ui(editor): add YAML document model, canvas model and run source logic
11 ui(editor): Add edit session, YAML format and quick fixes, run actions, tab strip
12 ui(editor): Add canvas graph model, Add step panel, issues popover, run bar
13 ui(canvas): Add layered layout, dependency edges, status styles, reference picker
14 ui(canvas): Add export, flow builder, upstream data panel, output capture
15 ui(canvas): Add node cards, launchers, inspector sections, client issues
16 ui(editor): Add visual editor, step inspector and YAML file panel
17 ui(lists): Open runs on the workflow page; add tab and schedule helpers
18 ui(workspace): Add workspace URL state, legacy redirects and run list links
19 ui(workspace): Add workspace data layer, run facts and Workflow panel
20 ui(workspace): Add run start controls, visible runs and reuse source
21 ui(workspace): Add data provider, bar identity, continuous panel and step list
22 ui(logs): Add run actions cluster, cached logs and log popout page
23 ui(workspace): Add Activity dock, step run view and workflow page
24 test(workspace): Add integration tests for the workspace shell and step view
25 ui(pages): Retire classic run pages and open runs on the workflow page
26 test(editor): Add integration tests for the editor, inspector and run overlay
27 test(ui): Add repository-wide guard tests and ESLint boundary rules
28 e2e: Add workspace helpers and fixtures, update existing specs
29 e2e: Add workspace specs, part 1 of 2
30 e2e: Add workspace specs, part 2 of 2
31 examples: Add the workspace demo workflows

Stack 2/31. Base branch: stack/01-g1. Review and merge in order; this PR builds on the previous one.
Size: 21 files changed, 898 insertions(+), 34 deletions(-).
Checked: Go build, vet and tests of the changed packages, on top of the previous PR.

Summary

This PR adds fields to the persisted run state: Continuous on a node, and Ephemeral, SkipHandlers and ReusedInputs on a run. It adds an ephemeral store to the DAG-run repository. It changes run timestamps to millisecond precision and bounds the size of an open status file.

Why

Selected-step runs and continuous runs need to be recorded in the run state before the runtime and the API can use them. PR 1 gives the samples store. PR 3 uses these fields in the runner, and PR 4 sets them from the API and the CLI.

Changes

  • Run state (internal/ir)
    • run_status.go: DAGRunStatus.Ephemeral, SkipHandlers, ReusedInputs; types ReusedInput and ReusedInputVia (run or cache). run_node.go: Continuous on the node state.
    • run_status_builder.go: options WithEphemeral, WithSkipHandlers, WithReusedInputs (copies the slice).
  • Timestamps (internal/cmn/stringutil)
    • FormatTime writes TimestampLayout with three fractional digits. New FormatTimeSeconds keeps second precision. ParseTime reads old and new values. Old status files load with no migration.
  • Ephemeral runs (internal/persis)
    • dagrun_repository_ephemeral.go: createStore, storeHolding, EphemeralRunTTL, ErrEphemeralUnsupported. A retry or child run follows its root run.
    • DAGRunCreateAttemptOptions.Ephemeral, DAGRunRepositoryOptions.EphemeralStore. Ephemeral runs emit no events and do not appear in listings (runstate.go, dagrun_repository_events.go).
    • NewFileDAGRunRepositoryWithEphemeral in persis/file.
  • Status file bound (persis/file/dagrun/attempt.go)
    • compactIfLargeLocked replaces an open status file larger than 1 MiB with the last record. It needs at least two appends between compactions.
    • Attempt.Dir() and AttemptDir() give the attempt directory, used for the samples store in PR 3.
  • Small fixes: queue_abort.go and artifact/store.go use the new format function. testutil/dagrun_repository.go gets a helper.

How to review

  1. Read the new fields in internal/ir/run_status.go and the builder options.
  2. Read internal/cmn/stringutil/stringutil.go (format and parse).
  3. Read dagrun_repository_ephemeral.go, then the changes in dagrun_repository.go.
  4. Read compactStatusLocked in attempt.go.

Pay most attention to these places:

  • Old status files with second-precision times must still load.
  • Compaction must never lose the last record, and must not loop when one record is larger than the limit.

Tests

  • internal/ir/run_status_test.go, run_status_builder_reused_test.go: millisecond times, old files, ReusedInputs round trip.
  • internal/cmn/stringutil/stringutil_test.go: format, fixed width, parse.
  • internal/persis/dagrun_repository_ephemeral_test.go: hidden from listings, retries follow the root, cleanup.
  • internal/persis/file/dagrun/attempt_compaction_test.go: compaction of an open file.

Screenshots

No visible UI change in this PR.
Run chips that show reused outputs appear in a later UI PR of the stack.

This was referenced Oct 28, 2026
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
Open
This was referenced Oct 28, 2026
Merged
Merged
@yohamta0
yohamta0 force-pushed the stack/02-g2 branch 2 times, most recently from 7ad917d to 1a3559c Compare October 30, 2026 06:44
SCcagg5 and others added 8 commits October 31, 2026 19:34
@SCcagg5
specs: Add spec 083 and the samples store (dagucloud#3381)
f37ca2e 
* Spec 083 and the samples store for continuous runs
* Satisfy linters in the samples store
* Fix samples store issues found in review
* Keep backfilled tiers and sample order complete
* Keep sample stores apart and streak memory flat
* Keep sample counts and order right after a crash
@SCcagg5 @yohamta0
Run state and IR support for continuous runs and step outputs
2baeb4e
@yohamta0
Keep run IDs and cleanup right across stores
037555a 
A run ID names one run across the history and ephemeral stores, a
hidden ephemeral attempt reports no status data instead of falling
through, a not-started ephemeral run is not removed, the recent status
limit is the same with ephemeral runs, and a queued status records an
ephemeral attempt.
@yohamta0
Keep status writes durable across compaction
21cf5ad 
A running attempt keeps appending to the compacted file, so its
directory entry is synced. A writer that could not reopen after a
compaction opens again on the next write instead of losing the rest
of the run.
@yohamta0
Compare manual action finish times by value
0555464 
Timestamps now carry milliseconds, so a status written before that
names the same finish time in another form. Compare the instants, so
that such a waiting run is not rejected as a changed manual action.
@yohamta0
Keep ephemeral artifacts and status writes intact
e6f389b 
An artifact sweep read run liveness from the history store only, so the
artifacts of an ephemeral run, which share the artifact root, looked
orphaned and were removed. The repository now passes its ephemeral
store to the sweep, which keeps the runs of both trees.

A compaction replaces the status file by rename, so another attempt
that held the old file open, such as a compare-and-swap beside the
runner, appended to a file no reader sees. A write now reopens the
path when it names another file than the open one.

An older worker formats a stored time without its milliseconds, so the
waiting report of a resumed run differed from the stored approval time
and the coordinator rejected it. A time without a fraction now matches
any time in the same second.
@yohamta0
Apply the ephemeral option to new root runs only
e4cdf14 
The option is documented to apply only to a new root run, but a retry
or a child run that carried it went to the ephemeral store even when its
root lived in the history, splitting the attempt from its root. Such
attempts now follow their root like any other.
@yohamta0
Keep an ephemeral run whose retry is starting
7263b7b 
The ephemeral cleanup lists ended runs and removes each one later. A
retry that starts in between creates an attempt that the removal did
not see: before the runner opens it, it has no status file, so the
latest attempt is still the ended one, and a dispatched attempt is
NotStarted, which is not active. The removal then deleted the run with
the new attempt, its logs and its work directory.

The cleanup now names the attempt it judged, and the store refuses the
removal under its lock when a newer attempt exists.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SCcagg5 @yohamta0