Repository navigation
[VL] Spark 4.1: Velox split function returns incorrect results with limit parameter (SPARK-49968) #11913
Description
Backend
VL (Velox)
Bug description
split('hello', '', 1) returns ["h"] in Velox but should return ["hello"]. This is a Velox C++ re2-based split implementation issue when the limit parameter is specified. Spark 4.1 added the limit parameter test via SPARK-49968.
Spark 4.1 only.
Parent issue: #11550
Impact
| Suite / Item | Status | spark40 | spark41 |
|---|---|---|---|
| GlutenCollationRegexpExpressionsSuite | Entire suite TODO (1 failure) | 🟢 | 🔴 |
| GlutenRegexpExpressionsSuite | .exclude("SPLIT") |
🟢 | 🔴 |
| VeloxStringFunctionsSuite | testWithSpecifiedSparkVersion skips 4.x |
🟢 | 🔴 |
References
- Apache Spark JIRA: SPARK-49968
Activity
Yao-MR commentedon Apr 20, 2026
i will take this issue
brijrajk commentedon May 29, 2026
Hi @Yao-MR, are you still planning to work on this? Happy to pick it up if you've moved on.
Yao-MR commentedon Jun 1, 2026
yes, the related change on velox side is in review, will move on here once the change on the velox is merged
rdtr commentedon Oct 3, 2026
The Velox fix (facebookincubator/velox#17327) was closed, so I've taken it over in facebookincubator/velox#19335, crediting @Yao-MR as a co-author. Once it lands, I'll pass
spark.sql.legacy.truncateForEmptyRegexSplitto Velox in Gluten.