Repository navigation
Epic: ModelOutputThunk structural cleanup #909
Description
Status note: description rewritten on 2026-06-05 to reflect the post-#1197 design. The original April 2026 description is preserved in this issue's edit history. Discussion and design rationale: #1197.
Goal
Reshape ModelOutputThunk's internal layout to make the public surface explicit, the private surface coherent, and copy semantics structural rather than per-field.
Areas
| # | Area | Issue | Status |
|---|---|---|---|
| 1 | Raw provider responses → typed mot.raw namespace |
#1215 | not started |
| 2 | telemetry_span / OTel ownership |
#1181 | approved, awaiting merge — closes Area 2 on merge |
| 3 | Computation logistics → _call / _gen sub-objects |
#1216 | not started |
| 4 | _thinking → public mot.thinking |
#1217 | not started |
| 5 | Public error / generate_log surface | #1191 | folded in; design captured in #1191 comment |
TL;DR
- Area 1 (raw responses) — move backend
_meta[...]keys (chat_response,oai_chat_response, etc.) onto a typedmot.raw: RawProviderResponsenamespace. Migratechat.py:_parseto switch onmot.raw.provider. - Area 2 (telemetry_span) — resolved by refactor(telemetry)!: move backend tracing onto plugin/hook pattern #1181 on merge; drops out of scope.
- Area 3 (computation logistics) — split the seventeen private attrs into two internal sub-objects:
mot._call(originating call, preserved across copies) andmot._gen(in-flight machinery, reset on copy). Add a guard that raises on copy-of-uncomputed-MOT. - Area 4 (
_thinking) — promote to publicmot.thinkingwith a property alias for_thinkingthrough one deprecation cycle. - Area 5 (Soft-fail errors on ModelOutputThunk are only reachable via a private attribute #1191) — give MOT a public surface for errors and the generate log: expose
mot.generate_logas a public property, and add a publicmot.errorchannel so callers can detect a soft-failed thunk without reading private state.
Sequencing
- refactor(telemetry)!: move backend tracing onto plugin/hook pattern #1181 lands first. Independent, already approved. Closes Area 2 and adds
_generation_idto Area 3's field set. - Area 5 (Soft-fail errors on ModelOutputThunk are only reachable via a private attribute #1191). Smallest. Closes Soft-fail errors on ModelOutputThunk are only reachable via a private attribute #1191.
- Area 1 (refactor(core): consolidate backend raw responses onto mot.raw namespace #1215). No dependency on Areas 3/4/5.
- Area 4 (refactor(core): promote mot._thinking to public mot.thinking #1217). Independent. Same release as Area 1 is fine.
- Area 3 (refactor(core): split MOT internal state into _call / _gen sub-objects #1216). Largest internal rename. Lands last.
Coordinated, not in scope
- feat: Phase 2 — validated streaming #1013 (Phase 2 streaming chunking) — Area 3's
_genshape will accommodate parsed-repr stream buffers; do not block on feat: Phase 2 — validated streaming #1013, which could remain a placeholder for months. - Standardize logits in MoT #123 (logits) — when implemented, logits go in
mot.generation(e.g.generation.logprobs), notmot.raw. Per ModelOutputThunk field refactor: partition fields into sub-structures #793 precedent. ModelOutputThunkshould not be aCBlock#269 (MOT-not-CBlock) — orthogonal. If acted on later, Area 1'smot.rawis unaffected.
Related bugs (discovered during audit, filed separately)
- fix(backends): unify raw-path token usage on mot.generation.usage; guard eval_count=None #1218 —
fix(backends): unify raw-path token usage on mot.generation.usage; guard eval_count=None - fix(streaming): use public mot.cancelled property in streaming.py:437 #1219 —
fix(streaming): use public mot.cancelled property in streaming.py:437 - Better handling of reasoning output from thinking models #1201 —
Messagehas nothinkingfield; reasoning silently dropped in multi-turn chat (coordinated with Area 4)
Cross-cutting constraints
_metaisCBlocksurface, not just MOT surface. This epic does not changeCBlock._meta; it only removes MOT's writes into the dict.- Public API is wider than field names suggest.
mot._thinking,mot._meta[<keys>], andmot._generate_logare read by code outsidebase.pytoday; renames need property aliases for one minor release. - Streaming validation locks in current shape:
cancel_generation(),mot._async_queuesingle-consumer semantics, andmot.cancelledare load-bearing forstream_with_chunking()(feat(stdlib): add stream_with_chunking() with per-chunk validation (#901) #942). - Mellea is pre-1.0 —
DeprecationWarningaliases for one minor cycle, then remove.
Metadata
Metadata
Assignees
Labels
Component type system, CBlock, @generative decorator, prompt templates, RichDocumentSamplingStrategy, SamplingResult, ModelOption, generation optionsCore abstractions: Context, MOT, SamplingStrategy, formatters, serializationStreaming chunks, events, per-chunk validationNew feature or requestHigh level EpicMedium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.
Activity
ajbozarth commentedon Apr 22, 2026
@jakelorocco I'm unsure if this fits into the deliverable for 0.5 or not, if you think I should tackle it this sprint I'll assign it to myself and add it to the iteration. Otherwise this will just sit until other work is done
psschwei commentedon Apr 23, 2026
I'm not sure if the areas that would be changed under this (or #793 for that matter) would overlap with changes coming out of #891 , but in the case that there were overlaps, I would think that #891 is a higher priority to land first.
ajbozarth commentedon Apr 23, 2026
Flagging design tension with #891 before this gets assigned.
Three specific collision points:
1.
_async_queue/ computation logistics fieldsSub-issue #901 (
stream_with_chunking()) is explicitly built around the single-consumer constraint on_async_queue, including cancellation behavior. If #909 restructures those internal fields first, thestream_with_chunking()implementation will need to adapt to whatever new shape they land in — doing them in the wrong order creates rework.2.
telemetry_spanin._metaIf #909 relocates
telemetry_span, it intersects with an open concern from the #891 review: when doesGENERATION_POST_CALLfire in the streaming path, and how doesTokenMetricsPluginsee it? Thestream_with_chunking()path bypassesBaseSamplingStrategyentirely, so where the span lives affects whether that concern is resolved cleanly or left dangling.3. Phase 2 of #891 vs. logistics grouping
Epic #891 scopes a second phase where the MOT owns chunking via a new
stream_parsed_reprmethod, driven by the need to relax the single-consumer constraint. Issue #909's "computation logistics" grouping touches the same internals. If #909 lands first without considering that shape, it may make the Phase 2 MOT API harder to design.Recommendation: hold #909 until #891's core sub-issues (#898–#901) are landed. Once
stream_with_chunking()is stable, the scope of #909's logistics grouping will be much clearer — and some of it may naturally align with or be superseded by the Phase 2 design.psschwei commentedon Apr 23, 2026
Moving this out of ready based on above.
Backlog probably isn't the right status (maybe we should have something like "Hold" or "Blocked", but that's a different discussion)
ajbozarth commentedon May 14, 2026
Moved this into next sprint as it is still blocked by #942 which even if it lands soon I probably won't have bandwidth to get to this by end of sprint. Should still get to it this release cycle though
ajbozarth commentedon Jun 3, 2026
Design proposal for this issue posted as Discussion #1197 to allow per-area threading. Each area, the appendix, and the open questions live in separate comments there.
If the proposed shape is approved, I'll convert this issue into an epic with sub-issues per area and fold #1191 in as Area 5.
ajbozarth commentedon Jun 5, 2026
Description rewritten on 2026-06-05 to reflect the post-#1197 design. The original April 2026 description has been superseded; design discussion and per-area open-question resolution lives in #1197.
The five areas are now scoped as separate sub-issues; this issue tracks the epic.
[-]ModelOutputThunk structural cleanup — ._meta partitioning, raw responses, _thinking[/-][+]Epic: ModelOutputThunk structural cleanup[/+]ajbozarth commentedon Jun 19, 2026
Status refresh (2026-06-19)
Quick housekeeping on the "Related bugs" list, since both have landed:
_meta["usage"]writes from all five backends' raw paths and migratesbudget_forcing_alg.pyto readmot.generation.usage). The additive half (populatingmot.generation.usagefrom raw paths for ollama/hf/watsonx) was folded into refactor(backends): move generate_from_raw hook firing into Backend base class #1264.Area 2 already closed by #1181 as planned. So of the original five Areas + three related bugs, Areas 2/related-bugs are done; Areas 1/3/4/5 remain.
One adjacent merge worth a flag: #1264 moved
generate_from_rawto a@finalwrapper and made backends implement_generate_from_raw. Doesn't change any of the Area designs structurally, but the migration target function name is now_generate_from_raw(notgenerate_from_raw) for Areas 1 and 5 follow-up PRs.ajbozarth commentedon Jun 19, 2026
Post-epic follow-up surfaced from Area 5 (#1191)
Area 5's design audited every
GenerateLogfield and found most are duplicated on the MOT or are migrating in the rest of the epic — table in the #1191 design comment. Onlyprompt,date,is_final_result, andextraare not covered elsewhere.A future deprecation issue would propose:
mot.generation.prompt(new field onGenerationMetadata)mot.generation.timestamp(new field onGenerationMetadata)mot._is_final_result(sampling-state, off the log)GenerateLog+m.validate(generate_logs=...)+_generate_logtogetherextraentirelyOut of scope for any single Area — touches every backend's
post_processing, sampling'sis_final_resultplumbing, and the publicvalidateAPI. To be filed as its own issue once the remaining Areas (#1215, #1216, #1217) have merged.