You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(core): collect replaceable blocks and share the streaming filter - #1026
First of three stacked PRs that add custom block renderers to EnrichedMarkdownText (renderers and resolveImageSource). This one is the C++ core and ships inert: nothing consumes it until the native PR and the JS PR on top of it.
Stack: this PR → native hosts → JS API, docs and example.
BlockKinds: one table per replaceable kind (image paragraph, <video> block, link-only paragraph) with its node matcher, its name and its streaming line rule. The collector, the annotator and the streaming filter all iterate the table, so adding a kind is one entry here plus a margin entry per host.
BlockSlots: collects the slots a renderer may replace and every image occurrence, in document order, with stable ids (<kind>:<fnv1a64 of url>:<n>; slots and occurrences are counted separately so inline images coming and going do not shift block ids). One walk stamps the active ids on the AST. Nothing inside a list item is a slot, because the hosts lay lists out as a single run of text.
StreamingFilter: the math, table, code fence and media rules that iOS, Android and the JS collector previously kept as three copies now live once. New rules: a trailing image, link or <video> line is withheld while its syntax is open (hidden mode waits for the next block), a bare URL on its own line stays withheld until the next line (it can always grow), and an inline ), so a link block can carry metadata such as a file size.
Behaviour changes for existing streaming users: a half-streamed  and the new streaming_filter_test (about 50 cases across math, tables, fences, media, links and inline tails).
Exercised end to end through the two PRs above on the iPhone 17 simulator and a Pixel 9 emulator.
PR Checklist
Code compiles and runs on iOS
Code compiles and runs on Android
Updated documentation/README if applicable (in the JS PR of this stack)
Ran example app to verify changes
E2E tests are passing
Required E2E tests have been added (if applicable)
Quoted code fences are not recognized because parseFenceMarker() accepts only leading spaces. For > ```md\n> ![a](https://e, Progressive filtering consequently treats the second line as media and deletes it, although it is literal fenced-code content. Make fence tracking container-aware so media trimming skips code inside blockquotes and lists while preserving the original offsets.
Complete reference links and images are truncated as unfinished inline syntax. With [r]: https://e.com defined earlier, trailing See [docs][r] becomes See [docs] even in Progressive mode because the scanner selects the final [r]. Likewise, ![logo][r] becomes ![logo], removing the image. Recognize resolved full and collapsed reference forms before applying the tail cut, and add regression cases with earlier reference definitions.
Checking only the last line releases unfinished multiline video syntax. For <video\n src="https://e, the last line matches neither a media classifier nor the inline-tail scanner, so the entire unfinished tag is returned in both modes. Multiline video tags are supported by tests/fixtures/html-blocks.md. Track the trailing media block across lines and withhold it from its opening line until the syntax closes.
The reason will be displayed to describe this comment to others. Learn more.
Went through the streaming filter with an eye on what it costs per tick, in particular for a long paragraph that arrives as a single line (no \n at all). Everything is linear in the document except the open-inline-syntax check, which is linear in links × last-line length; details inline, with measurements from a standalone build of StreamingFilter.cpp + BlockKinds.cpp at -O2. One correctness gap in the same area (soft line breaks) and one non-blocking nit about per-line copies.
The reason will be displayed to describe this comment to others. Learn more.
text.substr(close + 1) (and the same at line 227) allocates and copies the rest of the line just to ask whether it is blank, and this runs once per [ candidate from openInlineSyntaxStart. On a long single-line paragraph that is links × line length bytes of memcpy per tick. An index-based isBlankFrom(text, close + 1) does the same check with no allocation and usually stops at the first character.
Adds the C++ side of custom block renderers for EnrichedMarkdownText:
- BlockKinds: one table per replaceable kind (image paragraph, video block,
link-only paragraph) with its node matcher, naming and streaming line rule.
The collector, the annotator and the streaming filter iterate it, so a new
kind is one entry here plus a margin entry per host.
- BlockSlots: collects slots and image occurrences in document order with
stable ids (`<kind>:<fnv1a64 of url>:<n>`, slots and occurrences counted
separately) and stamps the active ones on the AST in one walk. Nothing
inside a list item is a slot, since the hosts lay lists out as text.
- StreamingFilter: the math, table, code fence and media rules iOS, Android
and the JS collector all shared by copy now live once. Trailing image,
link and video lines are withheld while their syntax is open (hidden mode
waits for the next block), a bare URL stays withheld until the next line,
and an inline image or link still being typed at the tail of the last
paragraph is cut so it never flashes as text.
- The parser keeps link titles, so a link block can carry metadata.
Behaviour changes for existing streaming users: a trailing partial
``
links now carry their title in the AST.
Tests: block_slots_test (eligibility, id stability, annotation, the hash
fixture shared with the web collector) and streaming_filter_test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The parser now keeps link titles, so the committed bundle has to be rebuilt
for scripts/test-web-bundle.mjs to agree with the golden AST dump.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rs-*
The CI workflow only triggers for pull requests into main, so the upper PRs
of the block renderers stack (#1027 onto #1026's branch, #1028 onto #1027's)
got no checks. Temporary; remove once the stack has merged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…aming
Review findings on the streaming classifier, all with regression cases:
- a closed bracket followed by prose (`[INFO] done`) is prose, not an open
link, so the line is no longer withheld;
- a link destination closes at its balanced `)`, so nested parentheses and
titles with parentheses classify correctly in both the line classifier
and the inline-tail scanner, which now share one reader;
- a closed autolink followed by text is prose, not a block;
- four or more spaces of indentation make the line indented code, which no
classifier matches and the tail scanner leaves alone;
- an escaped bracket or bang is literal text;
- code spans are matched by backtick-run length, so a double-backtick span
keeps its brackets.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The inline syntax helpers now follow CommonMark further:
- A label closes at its balanced `]`, skipping escapes and code spans, so
`[](v)` and `[a [b]](u)` classify as link paragraphs and an
outer link still typing around a closed image is cut, not shown as text.
- A destination may be `<pointy>` and may carry a quoted or parenthesised
title; parentheses inside the title no longer count as delimiters, so
`[a](u "t(")` is complete and `[a](u "t )` stays open.
- The tail scanner walks left over the `[` candidates until one is open.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A stream that stopped at `<`, `<vid`, `<htt` or a lone `!` showed those
characters as text for a tick before the line became a video, link or
image block. A strict prefix of `<video`, `<http` or `
eszlamczyk
left a comment
What/Why?
First of three stacked PRs that add custom block renderers to
EnrichedMarkdownText(renderersandresolveImageSource). This one is the C++ core and ships inert: nothing consumes it until the native PR and the JS PR on top of it.Stack: this PR → native hosts → JS API, docs and example.
BlockKinds: one table per replaceable kind (image paragraph,<video>block, link-only paragraph) with its node matcher, its name and its streaming line rule. The collector, the annotator and the streaming filter all iterate the table, so adding a kind is one entry here plus a margin entry per host.BlockSlots: collects the slots a renderer may replace and every image occurrence, in document order, with stable ids (<kind>:<fnv1a64 of url>:<n>; slots and occurrences are counted separately so inline images coming and going do not shift block ids). One walk stamps the active ids on the AST. Nothing inside a list item is a slot, because the hosts lay lists out as a single run of text.StreamingFilter: the math, table, code fence and media rules that iOS, Android and the JS collector previously kept as three copies now live once. New rules: a trailing image, link or<video>line is withheld while its syntax is open (hidden mode waits for the next block), a bare URL on its own line stays withheld until the next line (it can always grow), and an inline), so a link block can carry metadata such as a file size.Behaviour changes for existing streaming users: a half-streamed
 and the newstreaming_filter_test(about 50 cases across math, tables, fences, media, links and inline tails).PR Checklist
🤖 Generated with Claude Code