Repository navigation
Rebase to master - #12
Merged
Merged
Conversation
* remove arrow datasource * remove the dependency on log4j
* change structure of the project * remove datasource * clean unnecessary code
jackyromano
pushed a commit
to jackyromano/gazelle-jni
that referenced
this pull request
Apr 26, 2022
cleanup temporary files in header generation
lviiii
pushed a commit
to lviiii/gluten
that referenced
this pull request
Jul 25, 2022
* [ARROW-DATA-SOURCE-1] Reorganize the source code for the new repository organization * [ARROW-DATA-SOURCE-1] Make compiling succeed * [ADS-2] Fix RAM usage CI (apache#3) * Update tpch.yml * Update report_ram_log.yml * Update report_ram_log.yml * Delete github-ci-fix * Delete github-ci-fix2 * [Arrow-Data-Source-4]Add mkdocs.yml and update docs * [ADS-6] Add utility methods to check leaked Allocators/MemoryPools (apache#7) Close apache#6 * [NSE-51] Update ArrowWritableColumnVector * [ADS-9][Parquet] Parquet data source not replaced by default (apache#10) * [ADS-9][Parquet] Parquet data source not replaced by default * Code style * [ADS-13] Validate metric TaskMetrics.peakExecutionMemory for native SQL engine (apache#14) Closes apache#13 * [ADS-11]Modify title check and automatic link to Issues for PRs (apache#12) * [ADS-16] Upgrade Arrow version to 3.0.0 (apache#17) Closes apache#16 * Initialize new repo * Move arrow data source files to arrow-data-source directory * move native sql files to native-sql-engine folder * [NSE-86] Add root pom.xml; Remove native-sql-engine/core/ArrowWritableColumnVector.java (apache#88) * fix github actions Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> * fix building & CI Signed-off-by: Yuan Zhou <yuan.zhou@intel.com> Co-authored-by: Chen Haifeng <haifeng.chen@intel.com> Co-authored-by: zhixingheyi-tian <xiangxiang.shen@intel.com> Co-authored-by: Hongze Zhang <mailtozhz@126.com> Co-authored-by: HongW2019 <hong2.wang@intel.com> Co-authored-by: Hongze Zhang <hongze.zhang@intel.com> Co-authored-by: Rui Mo <rui.mo@intel.com>
rui-mo
pushed a commit
to rui-mo/gazelle-jni
that referenced
this pull request
Mar 21, 2023
rui-mo
pushed a commit
to rui-mo/gazelle-jni
that referenced
this pull request
Mar 23, 2023
kecookier
added a commit
to kecookier/gluten
that referenced
this pull request
Sep 25, 2023
Co-authored-by: zhaokuo03 <zhaokuo03@meituan.com>
zuochunwei
added a commit
that referenced
this pull request
Sep 25, 2023
* Add detailed timing for VeloxShuffleReader, which contains three steps: IPC、decompress、deserialize * improvement ShuffleReader metrics (#12) Co-authored-by: zuochunwei <zuochunwei@meituan.com> Co-authored-by: zhaokuo <zhaokuo_game@163.com>
wangxinshuo-bolt
added a commit
to wangxinshuo-bolt/incubator-gluten
that referenced
this pull request
Jul 24, 2026
* [CORE] Avoid duplicate getConfigValue definitions * [CORE] Add runtime creation options * [CORE] Add memory manager creation options * [CORE] Add memory manager lifecycle hooks * [BOLT] Move Bolt memory manager holder behind lifecycle hooks * [BOLT] Move parallel input iterator handling out of core * [BOLT] Move FLATTEN macro to Bolt utilities * [BOLT] Move native loader sources under cpp/bolt * [BUILD] Use shared core CMake for Bolt builds
taiyang-li
added a commit
that referenced
this pull request
Sep 10, 2026
* add bolt backend in gluten * fix bolt's native laoder * replace rowSize with rowCount * fix cpp build after sync origin/main with common patches merged * try to fix jvm build * finish build jvm * fix scala ut initializatio * Seperate memory code from cpp/core (#12) (#14) * [CORE] Avoid duplicate getConfigValue definitions * [CORE] Add runtime creation options * [CORE] Add memory manager creation options * [CORE] Add memory manager lifecycle hooks * [BOLT] Move Bolt memory manager holder behind lifecycle hooks * [BOLT] Move parallel input iterator handling out of core * [BOLT] Move FLATTEN macro to Bolt utilities * [BOLT] Move native loader sources under cpp/bolt * [BUILD] Use shared core CMake for Bolt builds * simplify shuffle related cpp/core change (#15) * [MINOR][CORE] move ConfigResolver/ShuffleReaderBase to cpp/bolt * [MINOR][CORE] remove cpp/core/utils/TestUtils.h * [CORE] change shuffle reader create factory * [MINOR][CORE] move ToStringVector to cpp/bolt * add shuffle metrics (#16) * add shuffle metrics * add shuffle metrics * [CORE] Remove RuntimeOptions task attempt plumbing (#17) * revert name in NativeMemoryManagerJniWrapper.create (#18) * fix gluten ut failed caused by createInputIterator and getResourceFilePath (#19) * Ignore bolt failed uts (#41) * ignore failed uts * wip * commit again * fix some conflicts * Fix failed ut: AllBoltConfiguration (#42) * close issue taiyang-li#29 * wip * Fix failed ut: MiscOperatorSuite (#43) * close issue taiyang-li#31 * fix conflicts of MetricsUtil.scala in bolt * Split shared library and remove MemoryManager name parameter (#20) * [CORE][BOLT] Dispatch JNI input iterators through Runtime * [CORE] Export shared JNI state accessors * [BOLT] Promote JVM-loaded native libraries to global visibility * [CORE][BOLT] Split Core and Bolt into independent JNI libraries * [CORE] Remove unused memory manager name plumbing * [BOLT] Populate operator metrics through the array protocol * [CORE][BOLT] Reduce native build divergence from community * [DOC] Record the Core diff-reduction design and snapshot * Remove doc * [BOLT] Restore original lazy vector metric semantics * [CORE] Restore community CMake and JNI entry signatures * [CORE] Remove JNI input iterator test * Fix failed ut: BoltUdfSuiteLocal (#44) * fix one ut * fix all failed uts * fix tidy * agents for bolt (#45) * fix failed uts in GlobalOffHeapMemorySuite.scala (#47) * [MINOR][CORE] Remove stale libhdfs install rule (#46) * [MINOR] Exclude Bolt char varchar mismatch tests (#48) * [CORE] Restore inline JNI state accessors (#49) * fix failed ut in BoltAggregateFunctionsDefaultSuite (#50) * Fix gluten sql query test suite (#55) * fix array.sql in GlutenSQLQueryTestSuite * fix array.sql in GlutenSQLQueryTestSuite * fix datetime-parsing-invalid.sql * fix raise_eror,assert_true and misc-functions.sql * fix named-function-arguments.sql * fix GlutenHiveSQLQuerySuite (#58) * fix BoltAdaptiveQueryExecSuite (#59) * cherry-pick shuffle offload enable and metrics commit (#32) * enable shuffle offload in BoltConfig * enable shuffle offload only when bolt memory manager is enabled * get metrics from operators for ShuffleReaderNode * reformat code * feat(bolt): add gluten flag to control bolt shuffle RowFormat * format code * feat(bolt): add gluten flag to control bolt shuffle RowFormat See merge request: !2012 * fix ut failed caused by shuffle offload * Fix uts in MiscOperatorSuite (#61) * ignore FullOuter in BroadcastNestLoopJoin * ignore test explode/posexplode function * ignore Optimize GetJsonObject(ToJson(NamedStruct)) * remove Check BoltResizeBatches is added on reused shuffle read * ignore last two uts * fix style * fix style * Format cpp/bash/py/cmake files to pass ci (#62) * format cpp files * fix py and bash * fix license header * format cmake * [MINOR] Fix Bolt shuffle resize UT (#64) * [BOLT] Fix Bolt build/test failures after rebasing upstream main (#65) * fix bolt test settings * fix build failure after rebase upstream * fix tidy * fix jvm build error * fix build issues * revert makefile * Fix failed velox ci (#66) * fix velox uts * fix bolt ut * [BOLT] Align JNI state lifecycle with Velox (#68) * [BOLT] Align JNI state lifecycle with Velox * Remove gflag namespace except * Refactor core jni input adapters (#69) * [CORE] Decouple JNI input adapters from Runtime * [BOLT] Use Core JNI input adapter factories * [CORE] Move shuffle stream JNI bridge to JniCommon * Add bolt ci pipeline (#67) * remove paimon profile * [BOLT] Add Bolt backend CI workflow for Spark 3.5 Add a single GitHub Actions workflow (bolt_backend_x86.yml) that builds the Bolt native libraries and runs the Spark 3.5 unit tests, mirroring the Velox backend CI but with one job. The test step aligns with 'make test_spark35' minus the paimon profile. Also route all Makefile Maven invocations through the 'build/mvn -ntp' wrapper instead of a bare 'mvn', and document the new CI in AGENTS.md. * revert agents.md * wip * fix docker image * fix docker image * fix docker image * fix docker image * fix docker image * fix docker image * fix docker image * wip * fix ci --------- Co-authored-by: liyang.127 <liyang.127@bytedance.com> * [BOLT] Adapt getShuffleWriterType to execution mode API * [MINOR] remove unused folly header in cpp/core/jni/JniCommon.cc (#12547) * remove bolt ci * remove useless codes * fix style * remove useless changes * fix java build failure * fix cpp build failure * fix ut * fix format * fix format * [BOLT] Make the Bolt C++ unit tests build and pass again (#71) The cpp tests had not been built since BUILD_TESTS was never enabled in CI, so several things had rotted: - cpp/bolt/tests/CMakeLists.txt was mangled by the C++ formatter run in b436c03 (source names split like `MemoryManagerTest .cc`); restore it. - Core tests linked bolt::bolt although they include no Bolt headers, and the target does not even exist yet when core/ is configured; drop it. Link glog before gluten so the test resolves gflags from the static archives instead of the copy exported by libgluten.so. - Bolt tests use -export-dynamic; hide the statically linked gflags with --exclude-libs so libgluten.so's own glog does not register its flags into the executable's registry a second time. - MemoryManagerTest / RuntimeTest were out of date with the 3-argument BoltMemoryManager constructor, Metrics(unsigned, std::string) and the createShuffleReader signature. - CreateBoltRuntime runs without a JVM, so keep spark.gluten.useBoltMemoryManager off there; enabling it makes BoltRuntime fetch the task attempt id via JNI. The shared-library build no longer runs `cmake --target install`: nothing consumes /usr/local/lib/libgluten.so, and Maven packages the natives from cpp/build/package/bolt/<platform>/<arch>/ which the build stages itself. The staging step now also drops the three libraries into cpp/build/releases/ so that directory keeps working without an install step. --------- Co-authored-by: kexianda <kexianda@bytedance.com> Co-authored-by: wangxinshuo-bolt <wangxinshuo.db@bytedance.com> Co-authored-by: Zhang Xiaofeng <zhangxffff@gmail.com> Co-authored-by: Hao Chen <850083756@qq.com> Co-authored-by: liyang.127 <liyang.127@bytedance.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.