Skip to content

feat: composite partial load matcher and load spec - #19791

Merged
clintropolis merged 1 commit into
apache:masterfrom
clintropolis:composite-partial-load-matcher
Jul 31, 2026
Merged

clintropolis merged 1 commit into
apache:masterfrom
clintropolis:composite-partial-load-matcher

Conversation

@clintropolis

Copy link
Copy Markdown
Member

Description

Adds a composite partial load matcher that does a union of matches from a list of child matchers, allowing for example partial loading both cluster groups and projections, or combining matchers to produce matches which would be impossible with a single matcher (like multiple include/exclude patterns which would interfere with each other if within the same wildcard pattern).

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1

Reviewed 11 of 11 changed files.

Validation: git diff --check passed.


This is an automated review by Codex GPT-5.6-Sol

final List<MatchResult> results = new ArrayList<>(matchers.size());
for (PartialLoadMatcher matcher : matchers) {
final MatchResult result = matcher.match(segment, baseLoadSpec);
if (result == null) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve valid unions when one projection branch is empty

Projection matchers return null when no configured projection exists, but this code treats that the same as an incompatible matcher and vetoes the entire composite. For example, an older segment lacking one requested projection will full-load under the default FULL_LOAD, even when another member validly selects cluster groups. Empty selections must be distinguished from genuinely incompatible members.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was planning to make improvements to the projection matchers which would solve this in a follow-up PR. Projection matchers should basically never return null, and instead if the matching projection is not present it should issue a load-spec that tells things to load the 'base' table projection (since all projections can be computed from the table itself if the projection isn't present).

@cecemei
cecemei self-requested a review July 30, 2026 18:19
@cecemei

cecemei commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

While reviewing this PR I've some thoughts on catalog, partial load match, etc... not feedback on the PR itself, just some future work thoughts:

  1. Canonical naming layer. Right now, clustered segments and projections each track their "grouping columns" differently (one uses a richer type-aware structure, the other a plain list of names). We can give both a shared, catalog-defined canonical name for these columns, so there's one consistent way to refer to the column this rule cares about.
  2. Fast-check for partial-load matching. Today, matching a rule against a segment requires re-deriving whether the segment's virtual column actually matches what the rule is asking for every single time. If a segment is already known to match the catalog's current schema, we could skip that work and just say "yes" immediately. If a segment doesn't match what the catalog currently says, that alone doesn't tell us the segment is wrong, it just means we don't know yet, so we still have to run the real check.
  3. Segment to catalog conformance check. To make idea 2 possible, each segment would need some way to know "does my schema still match what the catalog says today?" e.g., a stored fingerprint it can compare, similar to fingerprints rule matches.
  4. Fold into reindex/compaction. If a segment doesn't match the catalog, we could rewrite/reindex it in the background so it matches again over time.

@clintropolis
clintropolis merged commit c612a1e into apache:master Jul 31, 2026
28 checks passed
@clintropolis
clintropolis deleted the composite-partial-load-matcher branch July 31, 2026 05:02
@github-actions github-actions Bot added this to the 39.0.0 milestone Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants