Repository navigation
Conversation
Silthus
left a comment
Silthus
left a comment
There was a problem hiding this comment.
I still don't like that we are hammering the API endpoint on a delay basis and constantly requeuing when an LLM generation is running. I would rather re-explore the possibility to pass a one-time event secret and pick up the job again once the LLM generation is finished or after a certain timeout. It's not a constant polling, but rather a one-time resume based on a custom event that is fired that matches the running LLM job, which is then picked up and checked.
We also have a schedule that picks it up, and maybe we have intermediate schedule steps just to make sure we reschedule three times. Instead of hammering it every two seconds, let's reschedule at, let's say, 3, 6, and 9 minutes. At the 9-minute mark, it will abort the job with a timeout.
Before, just use the same machinery we have here. I don't think that changes the completion sequence. We need to see where in the stack downstream we need to adopt this, but the flow would be:
- Kick off a swing background LLM generation target as a VM.
- When a custom event comes in, match the VM state with its parameters that match the signature.
- Resume that VM state.
| if (outcome.kind === 'failed') { | ||
| return this.fail(result, outcome.code, outcome.message, startedAt) | ||
| } | ||
| if (outcome.kind === 'succeeded') { | ||
| return this.succeed(result, outcome.envelope, startedAt) | ||
| } | ||
| if (outcome.kind === 'unreachable') { | ||
| if ((metadata?.transportFailures ?? 0) >= MAX_TRANSPORT_FAILURES) { | ||
| return this.fail(result, 'gateway_unavailable', GATEWAY_UNAVAILABLE_MESSAGE, startedAt) | ||
| } | ||
| this.reschedule(result, invocation, { | ||
| generationId: outcome.generationId, | ||
| startedAt, | ||
| polls: (metadata?.polls ?? 0) + 1, | ||
| throttles: metadata?.throttles ?? 0, | ||
| transportFailures: (metadata?.transportFailures ?? 0) + 1, | ||
| }) | ||
| return result | ||
| } | ||
| if (outcome.throttled && (metadata?.throttles ?? 0) >= MAX_THROTTLE_BOUNCES) { | ||
| return this.fail(result, 'throttled', THROTTLED_MESSAGE, startedAt) | ||
| } | ||
|
|
||
| const accepted = outcome.generationId && !metadata?.generationId | ||
| this.reschedule( | ||
| result, | ||
| invocation, | ||
| { | ||
| generationId: outcome.generationId, | ||
| // The give-up budget belongs to the generation, so its clock starts when one is | ||
| // accepted rather than when a step that spent a minute in the throttle first tried. | ||
| startedAt: accepted ? DateTime.utc().toISO()! : startedAt, | ||
| polls: (metadata?.polls ?? 0) + 1, | ||
| throttles: (metadata?.throttles ?? 0) + (outcome.throttled ? 1 : 0), | ||
| transportFailures: metadata?.transportFailures ?? 0, | ||
| }, | ||
| outcome.waitSeconds | ||
| ) | ||
| return result |
There was a problem hiding this comment.
make this a switch statement instead
There was a problem hiding this comment.
Done in 9a8965b: the outcome dispatch in execute() is now a switch (outcome.kind) with one case per arm.
| * Whether this generation has used up its budget. Wall-clock time is the real bound; the iteration | ||
| * count is the backstop for a run whose reschedules land faster than the interval asks for. | ||
| */ | ||
| function isSpent(metadata: LlmGenerationMetadata): boolean { |
There was a problem hiding this comment.
rename to something more self speaking like: hasExhaustedInvocationBudget()
There was a problem hiding this comment.
Renamed to hasExhaustedInvocationBudget (9a8965b). With the v2 park-and-wake transport it is now a pure execution-count backstop; the wall-clock give-up lives in the backup-mark computation (secondsUntilNextBackupPoll).
| const params = invocation.queueParameters | ||
| if (params?.type !== 'llmGenerate') { | ||
| throw new Error('Bad invocation') | ||
| } |
There was a problem hiding this comment.
can we cast as ... the invocation and thus get a strongly typed sub object incl. metadata?
There was a problem hiding this comment.
Done in 9a8965b: the invocation is narrowed once to a typed LlmGenerationInvocation (queueParameters proven llmGenerate, queueMetadata typed as LlmGenerationMetadata), so the loose metadata cast is gone.
Adds the llmGenerate queue-parameter type and a generation service that submits to the workflows LLM endpoint and polls it: reschedule at +2s while pending, honor Retry-After on 429 up to ten bounces, give up terminally after 90 seconds or 50 polls, and map transport failures to a bounded retry before failing the step. Editor previews loop inline under a 25-second deadline instead of rescheduling. Registers template-workflow-llm (hidden) with prompt, output fields, and a six-model picker, and a mock whose placeholder envelope cannot be mistaken for a real generation. The v1 Postgres queue serializer learns to round-trip the new parameter type so a poll never resumes the VM on an empty stack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XmQejLq5BhgHigZJUK97C
…p ladder Replaces the two-second polling loop: an accepted generation now parks until its first backup mark and is normally woken early by the $workflows_llm_generation_finished internal event. The subscription matcher gains a branch that looks the parked job up by id, verifies the one-time wake token the CDP minted at submit against the job's own queue metadata, and pulls it to scheduled = NOW(). The event is a wake-only signal; the woken job still collects the result through the authenticated retrieve. Backup polls fire at 1, 3 and 5 minutes after acceptance as the safety net for a swallowed error anywhere along the wake path. A generation still pending at the last mark fails as deadline_exceeded. Transport failures keep their short bounded retry, and the budget now resets once a poll reaches the endpoint again, so blips at separate marks no longer accumulate. The iteration bound drops to a 20-execution backstop. Also folds in review feedback: the outcome dispatch is a switch, the budget check is named hasExhaustedInvocationBudget, and the invocation is narrowed once to a typed LlmGenerationInvocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XmBZpwtzi7uKicgAdyv9yA
b7a2522 to
9a8965b
Compare
Silthus
changed the title
Problem
The generation endpoint from the layer below has no caller: a workflow that reaches a Generate text step has no transport that submits the prompt and collects the result.
Layer 2 of the Generate text stack. It ships the CDP side: the queue-parameter type, the park-and-wake generation service, the subscription-matcher wake branch, and the hidden template.
Changes
llmGeneratecyclotron queue-parameter type and a generation service that submits to the workflows endpoint and parks the job.wake_tokenat submit and carries it onqueueMetadatathrough every reschedule, so a submit retried after a transport failure lands with the same token.$workflows_llm_generation_finished, the subscription matcher looks the parked job up by id, verifies the token against the job's own metadata, and setsscheduled = NOW().deadline_exceeded; a 20-execution backstop guards runaway reschedules.Retry-Afterfor up to ten bounces, then fails asthrottled. Throttle waits do not start the ladder clock.gateway_unavailable. The budget resets once a poll reaches the endpoint again, so blips at separate marks do not accumulate.isTest) loop inline under a 25-second deadline and report expiry as a preview notice, not a node failure. The wake path does not touch previews.template-workflow-llm(hidden): prompt with liquid templating, output fields dictionary, six-model picker defaulting togpt-5-mini. Its mock returns placeholder text that cannot be mistaken for a real generation.sendPushNotificationis left untouched.How did you test this code?
hogflow-executor.service.test.ts,hog_function.test.ts) that need the full test-database setup this environment lacks. Their diff is one dependency-stub line each, covered by the typecheck; CI runs them.Automatic notifications
Docs update
Docs, skills, and MCP updates ship in the top layer of this stack.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Same orchestration as the layer below: Opus 5 TDD implementation, three Fable 5 review lenses (9 findings, one blocker: transport errors escaping the status ladder as permanent step failures), Opus fix pass, Fable confirmation. Skills invoked: /tdd, /implement, /writing-tests, /writing-user-facing-copy (run-log strings), /writing-pr-descriptions, /stacking-prs.
v2 revision (Fable 5, single session, human-directed): the 2-second polling loop became park-and-wake after review. The subscription matcher gained the wake branch, the deadline copy changed with the ladder, and the three inline review comments are folded in (switch dispatch,
hasExhaustedInvocationBudgetrename, typed invocation narrowing). Skills invoked: /writing-tests, /writing-user-facing-copy, /stacking-prs, /writing-pr-descriptions. Spec deltas are recorded on issue #22 and in the updated blueprint artifact.