Skip to content

store_astra and store_atlas store metadata that cannot be read back into DocMetadata #3344

Open

Description

@ptlnextdoor

Description

store_astra and store_atlas write each chunk's meta as chunk.metadata.__dict__. Two things go wrong:

  • DocMetadata allows extra fields. Pydantic keeps them outside __dict__, so fields that upstream nodes add (for example anomaly_score) are not stored.
  • __dict__ holds every empty field as None. When a stored row is read back, DocMetadata(**meta) rejects None for fields such as nodeId, so the read raises a validation error.

store_milvus had the same read failure and was fixed in #3055 by storing DocMetadata.toDict(), which leaves empty fields out.

Steps to Reproduce

  1. Build DocMetadata(objectId='a', chunkId=0, parent='/a', isDeleted=False).
  2. Call DocMetadata(**metadata.__dict__), which is the dict the two nodes store.

Expected vs Actual Behavior

Expected: the stored meta reads back into DocMetadata and keeps the extra fields.
Actual: nodeId fails with "Input should be a valid string" for None, and extra fields are missing.

Severity

P2 - Medium

Platform

Linux, macOS, Windows (the code path does not depend on the platform)

Version / Environment

develop at a7f3770. Found by code reading and a local reproduction of the validation error, not against a running Astra or Atlas.

Activity

  1. github-actions commented on Nov 18, 2026

    @github-actions
    Contributor
    🤖 Internal: Discord sync marker

    Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

  2. added a commit that references this issue on Nov 18, 2026
    f0fdcfe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions