Repository navigation

[SPARK-57222][SDP] Implement SCD2 Batch Processor; Decompose affected rows - #56311

Closed

AnishMahto wants to merge 42 commits into

apache:masterfrom

AnishMahto:SPARK-57222-SCD2-decompose-affected-rows

Closed

[SPARK-57222][SDP] Implement SCD2 Batch Processor; Decompose affected rows#56311
AnishMahto wants to merge 42 commits into
apache:masterfrom
AnishMahto:SPARK-57222-SCD2-decompose-affected-rows

Commits on Jun 2, 2026
- SCD2 preprocess microbatch
  AnishMahto
  committed
  f579359
- add scaladocs
  AnishMahto
  committed
  3a34e55
- fixing indenting
  AnishMahto
  committed
  b77e557
- document decomposition tail concept
  AnishMahto
  committed
  437bfa4
- document event deduplication responisbilities and add test
  AnishMahto
  committed
  75adaa5
- core implementation
  AnishMahto
  committed
  35b9d51
- tests
  AnishMahto
  committed
  95d7280
- some test cleanup
  AnishMahto
  committed
  841bcd9
- cleanup tests
  AnishMahto
  committed
  53a3c5d
- cleanup
  AnishMahto
  committed
  7e28a20
- chrono order + decomposition impl
  AnishMahto
  committed
  766af3f
- tests for chrono order
  AnishMahto
  committed
  36a9a35
Commits on Jun 3, 2026
- PR feedbavk
  AnishMahto
  committed
  8fbca9f
- link TODO
  AnishMahto
  committed
  f2f6282
- Merge branch 'SPARK-57134-SCD2-preprocess-microbatch' into SPARK-57152-SCD2-find-affected-rows
  AnishMahto
  committed
  d7cc05d
- test cleanup
  AnishMahto
  committed
  8893202
- Merge branch 'SPARK-57152-SCD2-find-affected-rows' into SPARK-57222-SCD2-decompose-affected-rows
  AnishMahto
  committed
  0f72aad
- implement decomposeOutOfOrderRows
  AnishMahto
  committed
  aeee077
- comment cleanup
  AnishMahto
  committed
  f144ed7
- schema preservation on decomposition
  AnishMahto
  committed
  a92936b
- implement dropRedundantRowsPostDecomposition
  AnishMahto
  committed
  d06e581
Commits on Jun 4, 2026
Commits on Jun 5, 2026
- PR feedback
  AnishMahto
  committed
  b5abfad
- Merge branch 'SPARK-57152-SCD2-find-affected-rows' into SPARK-57222-SCD2-decompose-affected-rows
  AnishMahto
  committed
  fa6c8b0
Commits on Jun 7, 2026
- simplify post decomposition cleanup
  AnishMahto
  committed
  f10990d
Commits on Jun 8, 2026
- assert well formed rows post-decomposition
  AnishMahto
  committed
  8374dc3
- self-review
  AnishMahto
  committed
  422e1ec
- whitespace linting
  AnishMahto
  committed
  d12eefb
- Merge branch 'SPARK-57152-SCD2-find-affected-rows' into SPARK-57222-SCD2-decompose-affected-rows
  AnishMahto
  committed
  b14ca1c
- scaladoc phrasing
  AnishMahto
  committed
  84011d2
- Merge branch 'SPARK-57152-SCD2-find-affected-rows' into SPARK-57222-SCD2-decompose-affected-rows
  AnishMahto
  committed
  aef2ae0
- whitespace
  AnishMahto
  committed
  3b3d784
Commits on Jun 10, 2026
- Merge branch 'master' into SPARK-57222-SCD2-decompose-affected-rows
  AnishMahto
  committed
  1366d65
- linting
  AnishMahto
  committed
  0155906
- linting
  AnishMahto
  committed
  746bec7
- make variables private
  AnishMahto
  committed
  6c33b9e
Commits on Jun 11, 2026
- introduce Scd2IntervalColumns to bundle row interval columns
  AnishMahto
  committed
  89549f7
- Merge branch 'master' into SPARK-57222-SCD2-decompose-affected-rows
  AnishMahto
  committed
  7079934
Commits on Jun 30, 2026
- PR feedback
  AnishMahto
  committed
  9781895

Commits

Commits on Jun 2, 2026

Commits on Jun 3, 2026

Commits on Jun 4, 2026

Commits on Jun 5, 2026

Commits on Jun 7, 2026

Commits on Jun 8, 2026

Commits on Jun 10, 2026

Commits on Jun 11, 2026

Commits on Jun 30, 2026