Skip to content

core: side-channel frames for executor progress and task logs #590

Closed

Description

@stromanni

The wire half of the parent issue: new frames, and a way to introduce them without forcing scheduler and executors to upgrade together.

Frames

Added to ExecutorMessage in crates/taskito-core/src/worker/protocol.rs:

Progress { job_id: String, progress: i32 },
TaskLog  { job_id: String, task_name: String, level: String, message: String, extra_len: Option<usize> }, // + payload

extra_len follows the result_len convention already in Success: None is no payload, Some(0) an empty one. publish is a TaskLog at level result, so it needs no frame of its own.

Added to SchedulerMessage::Job, both #[serde(default)] so an executor built before this ignores them:

disabled_middleware: Vec<String>,
metadata: Option<String>,

SchedulerMessage::HelloAck grows #[serde(default)] capabilities: Vec<String>.

Neither addition bumps PROTOCOL_VERSION. Bumping it would reject every mismatched peer outright — deliberate for a real wire break, wrong here, and it would couple scheduler and executor releases in the one place #546 exists to decouple.

Client

ExecutorClient grows an outbound side-channel the bindings can hand to their dispatchers — the sink a running task reports through. Requirements:

  • Tasks never touch the shared Mutex<FrameWriter>; the sink is a bounded channel drained by the existing result thread, so a task in a progress loop cannot stall the writer or starve results.
  • Progress coalesces to latest-per-job under backlog. Logs cannot (each is data), so the queue drops oldest and counts what it dropped.
  • Nothing is emitted unless hello_ack advertised the capability. Dropping silently on an unadvertised scheduler is correct — the executor cannot know why, and failing the job over a progress call would be worse than the degradation this replaces.

Done when

executor_tests.rs covers, over MemoryTransport against a real RemoteDispatcher: a progress frame reaching the scheduler; a publish arriving as a TaskLog at level result; the job frame carrying disabled_middleware and metadata intact; a full outbound queue coalescing progress and dropping logs rather than blocking the task; and a scheduler advertising no capability receiving no such frames at all.

Parent issue carries the rationale.

Activity

  1. pratyush618 commented on Aug 2, 2026

    @pratyush618
    Contributor

    Done in #604 (5134fcf). ExecutorSideChannel::dropped_task_logs() exposes the shed counter, and a_full_outbound_queue_coalesces_progress_and_sheds_the_oldest_logs fills the queue deterministically — a gated write half rather than a race against the drain — asserting the flood never parks the task, that the oldest lines are shed, that the newest still arrives, and that 500 progress reports coalesce to at most two frames.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions