[SPARK-46979][SS] Add support for specifying key and value encoder separately and also for each col family in RocksDB state store provider

[anishshri-db]

What changes were proposed in this pull request?

Add support for specifying key and value encoder separately and also for each col family in RocksDB state store provider

Why are the changes needed?

This change allows us to specify encoder for key/values separately and avoid encoding additional bytes. Also, it allows us to set schemas/encoders for individual column families, which will be required for future changes related to transformWithState operator (listState/timer changes etc)

We are refactoring a bit here given the upcoming changes. so we are proposing to split key and value encoders.
Key encoders can be of 2 types:

• with prefix scan
• without prefix scan

Value encoders can also eventually be of 2 types:

• single value
• multiple values (used for list state)

And we now also allow setting schema and getting encoder for each column family.
So after the change, we can potentially allow something like this:

• col family 1 - with keySchema with prefix scan and valueSchema with single value and binary type
• col family 2 - with keySchema without prefix scan and valueSchema with multiple values

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit tests

[info] Run completed in 3 minutes, 5 seconds.
[info] Total number of tests run: 286
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 286, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No

[anishshri-db]

@HeartSaVioR - PTAL, thx !

[sahnib]

LGTM. Thanks for separating the encoders, this helps us avoid evolve the key/value encoders independently and use both Multi valued, and prefix key encoder.

[HeartSaVioR]

Only minors. Looks great in overall.

[HeartSaVioR]

(Maybe microbenchmark could tell that this could regress for default column family only - map lookup with carefully crafted lock operation in every op, though I'd rather not concern before we see actual regression.)

[anishshri-db]

Yea I didn't worry about it too much, given that the provider init likely happens once for long lived queries and where we can retain the use of the same provider on the same executor across m/batch executions.

[HeartSaVioR]

No, what I meant is to look up concurrent map per "every op" to figure out encoder, for existing stateful operators - previously it was just a reference to the field. But ops is relatively very cheap compared to commit as of now, so let's see.

[anishshri-db]

Ah ok - yea mainly didn't want to maintain 2 data structures for this. But if we find that its more expensive, then we can just split some of the logic for the default col family case

[HeartSaVioR]

+1 pending CI

[HeartSaVioR]

Thanks! Merging to master.