Repository navigation
[CH] Crash on gluten exit #5741
Description
Activity
ulimit -a
yarn@hello-dn119:/data/corefiles$ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1030220 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1000000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 65536 virtual memory (kbytes, -v) unlimited file locks (-x) unlimitedNot the limit problem.
The disk space is enough, not the disk problem.This
yarn.nodemanager.sleep-delay-before-sigkill.msparameter of yarn cause this.(gdb) bt #0 0x00007fa458537428 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fa45853902a in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007fa456a8fc15 in __gnu_cxx::__verbose_terminate_handler() () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #3 0x00007fa456a8de36 in __cxxabiv1::__terminate(void (*)()) () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #4 0x00007fa456a8de81 in std::terminate() () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #5 0x00007fa456a841cf in __cxa_pure_virtual () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #6 0x00007fa4568a76b0 in outputStream::print_cr(char const*, ...) () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #7 0x00007fa456a5d323 in VMError::report(outputStream*) () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #8 0x00007fa456a5ef1f in VMError::report_and_die() () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #9 0x00007fa4568a04e5 in JVM_handle_linux_signal () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #10 0x00007fa456892f48 in signalHandler(int, siginfo*, void*) () from /data6/hadoop/yarn/local/filecache/187148/jdk-8u391-linux-x64.tar.gz/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so #11 <signal handler called> #12 0x00007fa46b2bff2a in std::__1::allocator<Poco::AutoPtr<Poco::Notification>*>::allocate[abi:v15000](unsigned long) (__n=0, this=<optimized out>) at ../contrib/llvm-project/libcxx/include/__memory/allocator.h:107 #13 std::__1::allocator<Poco::AutoPtr<Poco::Notification>*>::allocate_at_least[abi:v15000](unsigned long) (__n=0, this=<optimized out>) at ../contrib/llvm-project/libcxx/include/__memory/allocator.h:119 #14 std::__1::allocate_at_least[abi:v15000]<std::__1::allocator<Poco::AutoPtr<Poco::Notification>*> >(std::__1::allocator<Poco::AutoPtr<Poco::Notification>*>&, unsigned long) (__n=0, __alloc=...) at ../contrib/llvm-project/libcxx/include/__memory/allocate_at_least.h:33 #15 std::__1::__allocate_at_least[abi:v15000]<std::__1::allocator<Poco::AutoPtr<Poco::Notification>*> >(std::__1::allocator<Poco::AutoPtr<Poco::Notification>*>&, unsigned long) (__n=0, __alloc=...) at ../contrib/llvm-project/libcxx/include/__memory/allocate_at_least.h:42 #16 std::__1::__split_buffer<Poco::AutoPtr<Poco::Notification>*, std::__1::allocator<Poco::AutoPtr<Poco::Notification>*>&>::__split_buffer (__cap=0, __start=0, this=<optimized out>, __a=...) at ../contrib/llvm-project/libcxx/include/__split_buffer:316 #17 std::__1::__split_buffer<Poco::AutoPtr<Poco::Notification>*, std::__1::allocator<Poco::AutoPtr<Poco::Notification>*> >::push_back[abi:v15000](Poco::AutoPtr<Poco::Notification>* const&) (this=0x7fa46c9879a0 <absl::synch_event+7664>, __x=<optimized out>) at ../contrib/llvm-project/libcxx/include/__split_buffer:559 #18 std::__1::deque<Poco::AutoPtr<Poco::Notification>, std::__1::allocator<Poco::AutoPtr<Poco::Notification> > >::__add_back_capacity ( this=0x7fa46c9879a0 <absl::synch_event+7664>) at ../contrib/llvm-project/libcxx/include/deque:2597 #19 0x0000000000000000 in ?? ()
Backend
CH (ClickHouse)
Bug description
But, when I gdb the core, get,
The core is not complete to trace it, need to dig why.
We first found the core on occur 2024.2.12.
Spark version
Spark-3.3.x
Spark configurations
No response
System information
No response
Relevant logs
No response