* add bolt backend in gluten * fix bolt's native laoder * replace rowSize with rowCount * fix cpp build after sync origin/main with common patches merged * try to fix jvm build * finish build jvm * fix scala ut initializatio * Seperate memory code from cpp/core (#12) (#14) * [CORE] Avoid duplicate getConfigValue definitions * [CORE] Add runtime creation options * [CORE] Add memory manager creation options * [CORE] Add memory manager lifecycle hooks * [BOLT] Move Bolt memory manager holder behind lifecycle hooks * [BOLT] Move parallel input iterator handling out of core * [BOLT] Move FLATTEN macro to Bolt utilities * [BOLT] Move native loader sources under cpp/bolt * [BUILD] Use shared core CMake for Bolt builds * simplify shuffle related cpp/core change (#15) * [MINOR][CORE] move ConfigResolver/ShuffleReaderBase to cpp/bolt * [MINOR][CORE] remove cpp/core/utils/TestUtils.h * [CORE] change shuffle reader create factory * [MINOR][CORE] move ToStringVector to cpp/bolt * add shuffle metrics (#16) * add shuffle metrics * add shuffle metrics * [CORE] Remove RuntimeOptions task attempt plumbing (#17) * revert name in NativeMemoryManagerJniWrapper.create (#18) * fix gluten ut failed caused by createInputIterator and getResourceFilePath (#19) * Ignore bolt failed uts (#41) * ignore failed uts * wip * commit again * fix some conflicts * Fix failed ut: AllBoltConfiguration (#42) * close issue taiyang-li#29 * wip * Fix failed ut: MiscOperatorSuite (#43) * close issue taiyang-li#31 * fix conflicts of MetricsUtil.scala in bolt * Split shared library and remove MemoryManager name parameter (#20) * [CORE][BOLT] Dispatch JNI input iterators through Runtime * [CORE] Export shared JNI state accessors * [BOLT] Promote JVM-loaded native libraries to global visibility * [CORE][BOLT] Split Core and Bolt into independent JNI libraries * [CORE] Remove unused memory manager name plumbing * [BOLT] Populate operator metrics through the array protocol * [CORE][BOLT] Reduce native build divergence from community * [DOC] Record the Core diff-reduction design and snapshot * Remove doc * [BOLT] Restore original lazy vector metric semantics * [CORE] Restore community CMake and JNI entry signatures * [CORE] Remove JNI input iterator test * Fix failed ut: BoltUdfSuiteLocal (#44) * fix one ut * fix all failed uts * fix tidy * agents for bolt (#45) * fix failed uts in GlobalOffHeapMemorySuite.scala (#47) * [MINOR][CORE] Remove stale libhdfs install rule (#46) * [MINOR] Exclude Bolt char varchar mismatch tests (#48) * [CORE] Restore inline JNI state accessors (#49) * fix failed ut in BoltAggregateFunctionsDefaultSuite (#50) * Fix gluten sql query test suite (#55) * fix array.sql in GlutenSQLQueryTestSuite * fix array.sql in GlutenSQLQueryTestSuite * fix datetime-parsing-invalid.sql * fix raise_eror,assert_true and misc-functions.sql * fix named-function-arguments.sql * fix GlutenHiveSQLQuerySuite (#58) * fix BoltAdaptiveQueryExecSuite (#59) * cherry-pick shuffle offload enable and metrics commit (#32) * enable shuffle offload in BoltConfig * enable shuffle offload only when bolt memory manager is enabled * get metrics from operators for ShuffleReaderNode * reformat code * feat(bolt): add gluten flag to control bolt shuffle RowFormat * format code * feat(bolt): add gluten flag to control bolt shuffle RowFormat See merge request: !2012 * fix ut failed caused by shuffle offload * Fix uts in MiscOperatorSuite (#61) * ignore FullOuter in BroadcastNestLoopJoin * ignore test explode/posexplode function * ignore Optimize GetJsonObject(ToJson(NamedStruct)) * remove Check BoltResizeBatches is added on reused shuffle read * ignore last two uts * fix style * fix style * Format cpp/bash/py/cmake files to pass ci (#62) * format cpp files * fix py and bash * fix license header * format cmake * [MINOR] Fix Bolt shuffle resize UT (#64) * [BOLT] Fix Bolt build/test failures after rebasing upstream main (#65) * fix bolt test settings * fix build failure after rebase upstream * fix tidy * fix jvm build error * fix build issues * revert makefile * Fix failed velox ci (#66) * fix velox uts * fix bolt ut * [BOLT] Align JNI state lifecycle with Velox (#68) * [BOLT] Align JNI state lifecycle with Velox * Remove gflag namespace except * Refactor core jni input adapters (#69) * [CORE] Decouple JNI input adapters from Runtime * [BOLT] Use Core JNI input adapter factories * [CORE] Move shuffle stream JNI bridge to JniCommon * Add bolt ci pipeline (#67) * remove paimon profile * [BOLT] Add Bolt backend CI workflow for Spark 3.5 Add a single GitHub Actions workflow (bolt_backend_x86.yml) that builds the Bolt native libraries and runs the Spark 3.5 unit tests, mirroring the Velox backend CI but with one job. The test step aligns with 'make test_spark35' minus the paimon profile. Also route all Makefile Maven invocations through the 'build/mvn -ntp' wrapper instead of a bare 'mvn', and document the new CI in AGENTS.md. * revert agents.md * wip * fix docker image * fix docker image * fix docker image * fix docker image * fix docker image * fix docker image * fix docker image * wip * fix ci --------- Co-authored-by: liyang.127 <liyang.127@bytedance.com> * [BOLT] Adapt getShuffleWriterType to execution mode API * [MINOR] remove unused folly header in cpp/core/jni/JniCommon.cc (#12547) * remove bolt ci * remove useless codes * fix style * remove useless changes * fix java build failure * fix cpp build failure * fix ut * fix format * fix format * [BOLT] Make the Bolt C++ unit tests build and pass again (#71) The cpp tests had not been built since BUILD_TESTS was never enabled in CI, so several things had rotted: - cpp/bolt/tests/CMakeLists.txt was mangled by the C++ formatter run in b436c03 (source names split like `MemoryManagerTest .cc`); restore it. - Core tests linked bolt::bolt although they include no Bolt headers, and the target does not even exist yet when core/ is configured; drop it. Link glog before gluten so the test resolves gflags from the static archives instead of the copy exported by libgluten.so. - Bolt tests use -export-dynamic; hide the statically linked gflags with --exclude-libs so libgluten.so's own glog does not register its flags into the executable's registry a second time. - MemoryManagerTest / RuntimeTest were out of date with the 3-argument BoltMemoryManager constructor, Metrics(unsigned, std::string) and the createShuffleReader signature. - CreateBoltRuntime runs without a JVM, so keep spark.gluten.useBoltMemoryManager off there; enabling it makes BoltRuntime fetch the task attempt id via JNI. The shared-library build no longer runs `cmake --target install`: nothing consumes /usr/local/lib/libgluten.so, and Maven packages the natives from cpp/build/package/bolt/<platform>/<arch>/ which the build stages itself. The staging step now also drops the three libraries into cpp/build/releases/ so that directory keeps working without an install step. --------- Co-authored-by: kexianda <kexianda@bytedance.com> Co-authored-by: wangxinshuo-bolt <wangxinshuo.db@bytedance.com> Co-authored-by: Zhang Xiaofeng <zhangxffff@gmail.com> Co-authored-by: Hao Chen <850083756@qq.com> Co-authored-by: liyang.127 <liyang.127@bytedance.com>

Support Spark Datasource V1 #58

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions