Skip to content

[Feature]: support multimodal message ingestion with transcript-first rawdata storage #4

Closed

Description

@starboyate

Is your feature request related to a problem? Please describe.
Memind currently assumes text-only messages. The current framework-neutral message model (ChatMessage) only carries role + String content, which means image and audio messages cannot enter the memory pipeline directly. In real chat applications, a large part of useful memory comes from screenshots, photos, voice notes, and other multimodal inputs. Today, users would need to pre-process those assets outside memind and manually convert them to text before ingestion.

Describe the solution you'd like
Add first-class multimodal message ingestion so memind can accept image and audio inputs, convert them into normalized text, and persist both the transcript and the original asset references in a way that fits the existing extraction pipeline.

Suggested scope for this issue:

  • extend the ingestion/message model to represent text, image, and audio content instead of only plain text
  • add a multimodal ingestion pipeline that can run OCR / captioning for images and ASR / transcription for audio
  • persist the normalized text output into rawdata so the existing rawdata -> item -> insight pipeline can continue to work without requiring a parallel retrieval path
  • store references to the original multimodal assets alongside the derived text, metadata, and processing provenance
  • introduce a pluggable asset storage abstraction so original image/audio resources can be stored either on local filesystem storage or in S3-compatible object storage
  • expose starter / configuration support so applications can choose storage backend and multimodal parsing providers

Recommended functional behavior:

  • image input: store original asset, extract OCR/caption text, then write normalized text into rawdata
  • audio input: store original asset, run transcription, then write normalized text into rawdata
  • rawdata should remain the canonical entry point for downstream extraction, with links back to the original asset location and media metadata

Out of scope for the first iteration:

  • full video ingestion
  • native binary retrieval without transcript
  • provider-specific UI concerns

Describe alternatives you've considered

  • Require upstream applications to convert image/audio to text before calling memind. This is simpler, but it pushes a core memory-ingestion responsibility to every integrator and makes multimodal behavior inconsistent across applications.
  • Only store image/audio metadata and asset references without transcript extraction. This preserves the files, but the current memory extraction and retrieval pipeline still cannot reason over the multimodal content.
  • Store binary payloads directly inside rawdata / database rows. This would complicate storage backends and is less portable than using asset references plus transcript-first indexing.

Additional context
This issue can act as an umbrella for follow-up sub-issues such as:

  • multimodal message model / API changes
  • asset storage SPI with local and S3 implementations
  • OCR / ASR provider abstraction
  • rawdata schema changes for media references and provenance
  • Spring Boot starter auto-configuration and examples
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions