An OPEN_STRUCT child field spec can declare a custom defaultNullValue, but only one of the four per-key storage layouts honors it. With null handling disabled, the same query can therefore return different values for the same key depending only on which segment answers it.
Current behavior
A document that does not carry the key reads as:
| Layout |
Effective default |
Resolved by |
| Key absent from the whole segment |
the declared child spec's defaultNullValue |
OpenStructDataSource#getValueFieldSpec |
| Materialized (dense) key |
standard dimension default of the stored type |
OpenStructColumnSplitter#writeDenseKeyColumn |
| Sparse key |
standard dimension default of the stored type |
SparseKeyDataSource |
| Consuming (mutable) key |
standard dimension default of the stored type |
MutableOpenStructIndex#allocateKeyColumn |
So for a declared STRING child with "defaultNullValue": "N/A", a segment in which the key never occurs reads N/A for every row, while a segment in which the key occurs at least once reads null for every other row.
Desired
Pick one effective-default rule and apply it across all four layouts. The natural choice is the declared child field spec, which OpenStructDataSource#getValueFieldSpec already returns, since it is the only rule that reflects what was configured.
The sealed and consuming paths are coupled: MutableOpenStructIndex deliberately mirrors OpenStructColumnSplitter so that a document's resolved value does not change when a consuming segment is sealed. They have to move together.
MapFilterOperator#trySparseJsonIndex already refuses the sparse JSON fast path for the key's declared default, so it needs no change.
Code sites
OpenStructColumnSplitter#writeDenseKeyColumn
MutableOpenStructIndex#allocateKeyColumn
SparseKeyDataSource
Each carries a TODO pointing at this issue.
An OPEN_STRUCT child field spec can declare a custom
defaultNullValue, but only one of the four per-key storage layouts honors it. With null handling disabled, the same query can therefore return different values for the same key depending only on which segment answers it.Current behavior
A document that does not carry the key reads as:
defaultNullValueOpenStructDataSource#getValueFieldSpecOpenStructColumnSplitter#writeDenseKeyColumnSparseKeyDataSourceMutableOpenStructIndex#allocateKeyColumnSo for a declared STRING child with
"defaultNullValue": "N/A", a segment in which the key never occurs readsN/Afor every row, while a segment in which the key occurs at least once readsnullfor every other row.Desired
Pick one effective-default rule and apply it across all four layouts. The natural choice is the declared child field spec, which
OpenStructDataSource#getValueFieldSpecalready returns, since it is the only rule that reflects what was configured.The sealed and consuming paths are coupled:
MutableOpenStructIndexdeliberately mirrorsOpenStructColumnSplitterso that a document's resolved value does not change when a consuming segment is sealed. They have to move together.MapFilterOperator#trySparseJsonIndexalready refuses the sparse JSON fast path for the key's declared default, so it needs no change.Code sites
OpenStructColumnSplitter#writeDenseKeyColumnMutableOpenStructIndex#allocateKeyColumnSparseKeyDataSourceEach carries a TODO pointing at this issue.