refactor: replace stream_with_chunking with single-task stream() #1440

Replace the two-task stream_with_chunking() + StreamChunkingResult with a single-task stream()/Streamer primitive consumed by a plain async for, and factor the inline chunk-boundary bookkeeping into a Chunker object. Chunking stays shared across validators in this issue; per-validator chunking and chunk-state ownership are a follow-up (#1441).

Builds on the POC in PR #1409.

Scope

Already prototyped (POC in PR #1409)

• stream() / Streamer — single generator on the caller's task, no background orchestration task. Durable state is the thin Streamer handle (failed_early, failure_reason, streaming_failures, full_text, final_validations, terminal mot).
• ModelOutputThunk.__aiter__ / __anext__ — wraps astream() in the async-iterator protocol so plain and validated streaming share one loop.
• Event emission via STREAMING_EVENT hook — full Phase 1 vocabulary (QuickCheckEvent, ChunkEvent, StreamingDoneEvent, FullValidationEvent, ErrorEvent, CompletedEvent) fired uniformly on the single task.

New in this issue

• Chunker — see design below. Factors the chunk-boundary state currently inline in the orchestrator into a reusable object.
• Rename the Phase 1 chunking strategies — SentenceChunker → SentenceChunking, WordChunker → WordChunking, ParagraphChunker → ParagraphChunking (the ChunkingStrategy ABC keeps its name). This frees Chunker for the new stateful object above. Rationale: feat(stdlib): add ChunkingStrategy ABC and built-in chunkers #899 specified these as ...Chunking; they shipped as ...Chunker in feat(stdlib): add ChunkingStrategy ABC and built-in chunkers #923 with no recorded rationale (silent implementation choice). ...Chunker (an agent noun) reads as "the thing that chunks," which fits the new stateful driver better than a stateless strategy — so the strategies are restored to their ...Chunking spec and Chunker names the driver. Note: these names are exported (__all__) and used across docs/examples/streaming/, tests, and docs — this is a breaking rename of public names, with all call sites updated (consistent with this issue's hard break, below).
• __aiter__ single-consumer guard — __aiter__/__anext__ make MOT look like a normal async iterator, but a second consumer splits the same stream rather than starting an independent one. Add a real guard: mark the thunk consumed on first iteration, raise if a second consumer starts. (Full parallel-consumer support — buffering, per-consumer cursors, late-joiner semantics — is out of scope unless a concrete need appears; the guard is enough for now.)
• Events example — an example demonstrating an events plugin that consumes STREAMING_EVENT and surfaces streaming events for multiple parallel streams. The plugin may live in the example or in src depending on implementation. This is the primary user-facing deliverable for event consumption (hooks are the interface; the example is how users learn to consume them).
• Hard break, no deprecation shim — stream() replaces stream_with_chunking() outright. A shim would keep the old orchestrator alive alongside the new one, reintroducing the complexity this issue removes; the API is new enough to break directly.

Constraints

• Plain strings only. stream() yields str chunks. The typed-chunk / multi-modal envelope belongs to stream_parsed_repr (feat(core): stream_parsed_repr — streaming parsed representation on the MOT #1442); mixing it in here means changing the same surface twice.

This issue's chunking scope

Exactly one Chunker on the Streamer, fed by the single chunking= param; all validators validate the same chunks (as today). Per-validator chunking — each Requirement carrying its own strategy, and the resulting chunk-state ownership question — is #1441. The Chunker carries its own small state, so that follow-up relocates/multiplies it without an internal reshape.

Chunker class

Factors the chunk-boundary bookkeeping that currently lives inline in _orchestrate_streaming / _drive (accumulated scanning, prev_chunk_count, withheld-fragment handling) into a dedicated object. It is the stateful counterpart to a ChunkingStrategy: the strategy is stateless "how to split," the Chunker holds "how far through this stream we are."

Shape

• __init__(strategy: ChunkingStrategy) — holds the (stateless) strategy.
• feed(delta: str) -> list[str] — takes one stream delta, returns any new complete chunks, holding the trailing fragment between calls.
• A flush/finalize method for the trailing fragment at stream end.

State division

The Chunker holds only the pending fragment, not the full accumulated text. The Streamer keeps accumulated for its own full_text needs and feeds deltas to the Chunker.

Implementation approach — reuse split(), don't reimplement boundaries

Chunker.feed() should call the existing stateless ChunkingStrategy.split() on the pending fragment, not reimplement boundary detection:

feed(delta):
    self._pending += delta
    chunks = self._strategy.split(self._pending)
    self._pending = <tail after last returned chunk>
    return chunks

split() already withholds the trailing fragment correctly, so feeding it the pending fragment reuses the exact boundary logic that already works. The only new logic is computing the tail carried forward.

Behavioral-change risk: chunking on the delta, not full accumulated

Today split() is called on the full accumulated every delta (O(n) re-scan). Moving to incremental delta input is a real behavioral change, not a pure refactor. Risk is confined to boundaries that span the delta seam:

• Sentence — boundary is punctuation + whitespace; the . and the following space can arrive in different deltas.
• Word — trailing partial word split across deltas.
• Paragraph — \n{2,} is greedy; the newlines can split across deltas, and a prefix (\n\n) must not be emitted as a complete boundary before a possible third \n arrives.

Reusing split() on the accumulated pending fragment (rather than delta-native detection) sidesteps most of this, because split()'s withhold-the-tail contract already handles incomplete trailing boundaries.

Required test: delta-invariance — for any way a text is sliced into deltas, feeding the slices to Chunker yields the same chunks as one split(full_text). Fixtures: 1 delta, per-character deltas, random splits, plus the three seam cases above. The test/core/test_astream_mock.py mock gives deterministic delta control.