Repository navigation
[Feature]: support multimodal message ingestion with transcript-first rawdata storage #4
Description
Is your feature request related to a problem? Please describe.
Memind currently assumes text-only messages. The current framework-neutral message model (ChatMessage) only carries role + String content, which means image and audio messages cannot enter the memory pipeline directly. In real chat applications, a large part of useful memory comes from screenshots, photos, voice notes, and other multimodal inputs. Today, users would need to pre-process those assets outside memind and manually convert them to text before ingestion.
Describe the solution you'd like
Add first-class multimodal message ingestion so memind can accept image and audio inputs, convert them into normalized text, and persist both the transcript and the original asset references in a way that fits the existing extraction pipeline.
Suggested scope for this issue:
- extend the ingestion/message model to represent text, image, and audio content instead of only plain text
- add a multimodal ingestion pipeline that can run OCR / captioning for images and ASR / transcription for audio
- persist the normalized text output into
rawdataso the existing rawdata -> item -> insight pipeline can continue to work without requiring a parallel retrieval path - store references to the original multimodal assets alongside the derived text, metadata, and processing provenance
- introduce a pluggable asset storage abstraction so original image/audio resources can be stored either on local filesystem storage or in S3-compatible object storage
- expose starter / configuration support so applications can choose storage backend and multimodal parsing providers
Recommended functional behavior:
- image input: store original asset, extract OCR/caption text, then write normalized text into
rawdata - audio input: store original asset, run transcription, then write normalized text into
rawdata rawdatashould remain the canonical entry point for downstream extraction, with links back to the original asset location and media metadata
Out of scope for the first iteration:
- full video ingestion
- native binary retrieval without transcript
- provider-specific UI concerns
Describe alternatives you've considered
- Require upstream applications to convert image/audio to text before calling memind. This is simpler, but it pushes a core memory-ingestion responsibility to every integrator and makes multimodal behavior inconsistent across applications.
- Only store image/audio metadata and asset references without transcript extraction. This preserves the files, but the current memory extraction and retrieval pipeline still cannot reason over the multimodal content.
- Store binary payloads directly inside
rawdata/ database rows. This would complicate storage backends and is less portable than using asset references plus transcript-first indexing.
Additional context
This issue can act as an umbrella for follow-up sub-issues such as:
- multimodal message model / API changes
- asset storage SPI with local and S3 implementations
- OCR / ASR provider abstraction
rawdataschema changes for media references and provenance- Spring Boot starter auto-configuration and examples
Activity