Skip to content

OPEN_STRUCT: declared child defaultNullValue is honored only for keys absent from the whole segment #19466

Description

@Jackie-Jiang

An OPEN_STRUCT child field spec can declare a custom defaultNullValue, but only one of the four per-key storage layouts honors it. With null handling disabled, the same query can therefore return different values for the same key depending only on which segment answers it.

Current behavior

A document that does not carry the key reads as:

Layout Effective default Resolved by
Key absent from the whole segment the declared child spec's defaultNullValue OpenStructDataSource#getValueFieldSpec
Materialized (dense) key standard dimension default of the stored type OpenStructColumnSplitter#writeDenseKeyColumn
Sparse key standard dimension default of the stored type SparseKeyDataSource
Consuming (mutable) key standard dimension default of the stored type MutableOpenStructIndex#allocateKeyColumn

So for a declared STRING child with "defaultNullValue": "N/A", a segment in which the key never occurs reads N/A for every row, while a segment in which the key occurs at least once reads null for every other row.

Desired

Pick one effective-default rule and apply it across all four layouts. The natural choice is the declared child field spec, which OpenStructDataSource#getValueFieldSpec already returns, since it is the only rule that reflects what was configured.

The sealed and consuming paths are coupled: MutableOpenStructIndex deliberately mirrors OpenStructColumnSplitter so that a document's resolved value does not change when a consuming segment is sealed. They have to move together.

MapFilterOperator#trySparseJsonIndex already refuses the sparse JSON fast path for the key's declared default, so it needs no change.

Code sites

  • OpenStructColumnSplitter#writeDenseKeyColumn
  • MutableOpenStructIndex#allocateKeyColumn
  • SparseKeyDataSource

Each carries a TODO pointing at this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is not working as expectednull supportRelated to NULL value handlingqueryRelated to query processing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions