Skip to content

[VL] Upgrade velox to 2023.3.29 - #1231

Merged
zhejiangxiaomai merged 1 commit into
apache:mainfrom
zhejiangxiaomai:main_rebase_327
Apr 1, 2023
Merged

zhejiangxiaomai merged 1 commit into
apache:mainfrom
zhejiangxiaomai:main_rebase_327

Conversation

@zhejiangxiaomai

@zhejiangxiaomai zhejiangxiaomai commented Mar 29, 2023 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  1. velox change memory pool API. we need call addChild to create memory pool to allocate.
  2. planFragment constructor change a little bit.
  3. setMemoryUsageTracker was removed by velox.
    corresponding velox branch is https://github.com/zhejiangxiaomai/velox/tree/main_rebase_329
  4. add " cmake --install xsimd-build/ cmake --install gtest-build/" in build_velox.sh. because velox move xsimd/gtest into resolve_dependency module.

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[Gluten-${ISSUES_ID}] ${detailed message}

See also:

@zhejiangxiaomai zhejiangxiaomai changed the title [VL] Upgrade velox to 3.27 [VL] Upgrade velox to 2023.3.27 Mar 29, 2023
@zhejiangxiaomai

zhejiangxiaomai commented Mar 29, 2023 •

Copy link
Copy Markdown
Contributor Author

get a core when run tpch.

spark.sql(" select s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment from part, supplier, partsupp, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and p_size = 15 and p_type like '%BRASS' and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'EUROPE' and ps_supplycost = ( select min(ps_supplycost) from partsupp, supplier, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'EUROPE') order by s_acctbal desc, n_name, s_name, p_partkey limit 1 ").show(1, false)

(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f7750fde859 in __GI_abort () at abort.c:79
#2 0x00007f775036af2f in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#3 0x00007f7750caa683 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#4 0x00007f7750afaf4e in JVM_handle_linux_signal () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#5 0x00007f7750aee0ac in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#6
#7 0x0000000000000000 in ?? ()
#8 0x00007f74782faf81 in facebook::velox::AlignedBuffer::freeToPool (this=0x7f7350025a30) at /root/gluten/cpp/../ep/build-velox/build/velox_ep/velox/buffer/Buffer.h:517
#9 0x00007f747824a48a in facebook::velox::Buffer::release (this=0x7f7350025a30) at /root/gluten/cpp/../ep/build-velox/build/velox_ep/velox/buffer/Buffer.h:75
#10 0x00007f747824a4e0 in facebook::velox::intrusive_ptr_release (buffer=0x7f7350025a30) at /root/gluten/cpp/../ep/build-velox/build/velox_ep/velox/buffer/Buffer.h:278
#11 0x00007f7478250fa7 in boost::intrusive_ptrfacebook::velox::Buffer::~intrusive_ptr (this=0x7f73500259f8, __in_chrg=) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:98
#12 0x00007f747c883923 in facebook::velox::(anonymous namespace)::VeloxToArrowBridgeHolder::~VeloxToArrowBridgeHolder (this=0x7f73500259d0, __in_chrg=) at ../../velox/vector/arrow/Bridge.cpp:37
#13 0x00007f747c883b54 in facebook::velox::(anonymous namespace)::releaseArrowArray (arrowArray=0x7f7350025970) at ../../velox/vector/arrow/Bridge.cpp:170
#14 0x00007f747c883ab2 in facebook::velox::(anonymous namespace)::releaseArrowArray (arrowArray=0x7f735004e440) at ../../velox/vector/arrow/Bridge.cpp:155
#15 0x00007f769c015d76 in Java_org_apache_arrow_c_jni_JniWrapper_releaseArray () from /tmp/jnilib-8155710934442135310.tmp
#16 0x00007f7739018427 in ?? ()
#17 0x00007f76764f09a0 in ?? ()
#18 0x00007f7675df0a08 in ?? ()
#19 0x0000000000000000 in ?? ()
@zhztheplayer could you please help me?

@zhztheplayer zhztheplayer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread cpp/velox/memory/VeloxMemoryPool.cc Outdated
Comment thread cpp/velox/memory/VeloxMemoryPool.cc Outdated
Comment thread cpp/velox/memory/VeloxMemoryPool.cc Outdated
@zhejiangxiaomai
zhejiangxiaomai marked this pull request as ready for review March 29, 2023 13:56
@zhejiangxiaomai zhejiangxiaomai changed the title [VL] Upgrade velox to 2023.3.27 [VL] Upgrade velox to 2023.3.29 Mar 29, 2023
@kyligence-git

Copy link
Copy Markdown
Contributor

Can one of the admins verify this patch?

@zhejiangxiaomai
zhejiangxiaomai force-pushed the main_rebase_327 branch 4 times, most recently from cb8c842 to 5654a54 Compare March 31, 2023 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants