Description
Many source docs (Confluence, incident/architecture docs) embed diagrams/images that hold high-value knowledge — especially architecture diagrams, which encode system relationships (the #1 context class for incident resolution). Current text-only extraction drops them.
Primary approach (preferred): preserve, don't rewrite. Instead of "read the diagram and convert it to entities" (interpret-at-write-time, lossy), extract the raw images as-is, store them as assets in the block, have the relevant entity point to the image path, and serve the image to the multimodal consuming agent on-demand. The agent reads the original diagram at query time — zero information loss, interpretation deferred to the consumer, and it fits cache/router + progressive disclosure (#75).
Vision models are used only lightly — to generate a short caption/alt-text for retrievability, and (optionally) to classify ambiguous survivors — not to replace the image.
Why preserve-and-serve beats vision-rewrite
- Zero loss: the original diagram survives, not a lossy text summary.
- Same diagram interpreted per-question by the agent, not baked into one write-time interpretation.
- Cheaper/deterministic: pulling images from a PDF/PPTX is mechanical, no vision API at extraction.
- Consuming agents are multimodal anyway.
Extraction mechanics (per format — all mechanical, no AI)
- PDF: PyMuPDF (
page.get_images() / extract_image()) or pdfimages (poppler).
- DOCX / PPTX: ZIP archives — images in
word/media/ and ppt/media/.
- HTML / Confluence export:
<img> tags — linked (download) or base64-embedded (decode).
- Wrinkle: raster images extract cleanly; vector diagrams (native PPTX shapes, PDF vector paths) have no image stream — render the page/region to a raster instead.
MCP transport
Filtering — codified-first, vision optional
Decision of which images to keep is codified first, not vision-driven:
- Extract ALL images mechanically (no decision yet).
- Dedup by hash + frequency — an image repeated on most/all pages is chrome (e.g. a per-page logo): identical bytes → hash-match → high frequency → drop. Appears on only 1–2 pages → content → keep. Frequency is the discriminator. No vision needed — this is how the per-page-logo case is handled.
- Size threshold — drop tiny images (icons, bullets).
- Vision only as an optional light second pass on the few ambiguous survivors — classify "diagram/screenshot vs decorative." Runs on a handful, not every image.
Acceptance Criteria
Out of Scope
- Lossy vision-to-text rewrite as the primary mechanism (vision used only for caption hook + ambiguous classify)
- OCR-only text recovery
Description
Many source docs (Confluence, incident/architecture docs) embed diagrams/images that hold high-value knowledge — especially architecture diagrams, which encode system relationships (the #1 context class for incident resolution). Current text-only extraction drops them.
Primary approach (preferred): preserve, don't rewrite. Instead of "read the diagram and convert it to entities" (interpret-at-write-time, lossy), extract the raw images as-is, store them as assets in the block, have the relevant entity point to the image path, and serve the image to the multimodal consuming agent on-demand. The agent reads the original diagram at query time — zero information loss, interpretation deferred to the consumer, and it fits cache/router + progressive disclosure (#75).
Vision models are used only lightly — to generate a short caption/alt-text for retrievability, and (optionally) to classify ambiguous survivors — not to replace the image.
Why preserve-and-serve beats vision-rewrite
Extraction mechanics (per format — all mechanical, no AI)
page.get_images()/extract_image()) orpdfimages(poppler).word/media/andppt/media/.<img>tags — linked (download) or base64-embedded (decode).MCP transport
Filtering — codified-first, vision optional
Decision of which images to keep is codified first, not vision-driven:
Acceptance Criteria
Out of Scope