[VL] Spark 4.x: Enable Structured Streaming test suites #11911

Backend

VL (Velox)

Bug description

20 Structured Streaming test suites are disabled (TODO) for both Spark 4.0 and 4.1. Additionally, 3 already-enabled streaming suites have 11 excludes related to streaming API changes.

Goal: Enable SS test suites to run with GlutenPlugin loaded, allowing fallback to vanilla Spark where needed.

Parent issue: #11550

Disabled suites (20)

19 suites use GlutenSQLTestsTrait, 1 uses GlutenTestsCommonTrait.

Excludes in already-enabled suites (from #11400)

• GlutenStreamRealTimeModeAllowlistSuite (3 excludes)
• GlutenStreamRealTimeModeE2ESuite (7 excludes)
• GlutenStreamRealTimeModeSuite (1 exclude)
• SPARK-53942 stateful shuffle partitions (2 excludes)

Root causes

1. Plan assertion failures — GlutenPlugin replaces ShuffleExchangeExec with ColumnarShuffleExchangeExec, etc. Tests asserting specific plan nodes fail.
2. Checkpoint resource loading — Golden files contain plan structures that don't match Gluten-transformed plans.
3. Spark 4.1 streaming API changes — SPARK-53941 (AQE), SPARK-53233 (package refactor), etc.

Notes

• GlutenStreamingQueryHashPartitionVerifySuite uses GlutenTestsCommonTrait intentionally — switching to GlutenSQLTestsTrait causes diamond inheritance conflict with StreamTest. The actual issue is that getWorkspaceFilePath requires SPARK_HOME.
• Suggested order: start with [DISCUSS] Any ideas about unified memory management for different native engines? #1-3 (simple), then Enable Velox's calculation for TPC-H Q6 #4-16 per suite, RocksDB variants (Add a fake result for easier debug #18-20) follow parent suites.