Repository navigation
[VL] can not write hive table on HDFS #5879
Description
Description
when add the config "spark.gluten.sql.native.writer.enabled true"，and an error occurred："The file path is not local when writing data with parquet format in velox runtime!"
Does velox not support writing hive table on HDFS now？
void VeloxParquetDatasource::init(const std::unordered_map<std::string, std::string>& sparkConfs) { if (strncmp(filePath_.c_str(), "file:", 5) == 0) { sink_ = dwio::common::FileSink::create(filePath_, {.pool = pool_.get()}); } else { throw std::runtime_error("The file path is not local when writing data with parquet format in velox runtime!"); } ......... }
Activity
philo-he commentedon May 27, 2024
@RaoZhiRou-Z, did you do the test with gluten main branch?
RaoZhiRou-Z commentedon May 27, 2024
yes，the main branch
[-][VH] can not write hive table on HDFS[/-][+][VL] can not write hive table on HDFS[/+]philo-he commentedon May 27, 2024
@RaoZhiRou-Z, it seems a bug. I created one PR to fix it: #5881. Please take a review.
RaoZhiRou-Z commentedon May 29, 2024
thanks，I will have a try
RaoZhiRou-Z commentedon Jun 6, 2024
@philo-he I have tried the bug-fix，and there is another core error，the core stack shows that：
#0 __GI_raise (sig=sig@entry=6) at /root/work/deck/devel/toolchain/glibc-2.33/signal/raise.c:49
#1 0x00007f6143cfa527 in __GI_abort () at abort.c:79
#2 0x00007f61435ec089 in os::abort(bool) () from /home/work/zjh/env/jdk/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so
#3 0x00007f61437b5363 in VMError::report_and_die() () from /home/work/zjh/env/jdk/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so
#4 0x00007f61435f64e5 in JVM_handle_linux_signal () from /home/work/zjh/env/jdk/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so
#5 0x00007f61435e8f48 in signalHandler(int, siginfo, void) () from /home/work/zjh/env/jdk/jdk1.8.0_391/jre/lib/amd64/server/libjvm.so
#6
#7 0x00007f5ffc54e52a in facebook::velox::dwio::common::FileSink::writeWithLogging(std::vector<facebook::velox::dwio::common::DataBuffer<char, void>, std::allocator<facebook::velox::dwio::common::DataBuffer<char, void> > >&) ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libvelox.so
#8 0x00007f5ffc54f102 in facebook::velox::dwio::common::FileSink::write(facebook::velox::dwio::common::DataBuffer<char, void>) ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libvelox.so
#9 0x00007f5ffc3c9306 in facebook::velox::parquet::ArrowDataBufferSink::Close() ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libvelox.so
#10 0x00007f5ffc3c525a in facebook::velox::parquet::Writer::close() ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libvelox.so
#11 0x00007f5ffe1ce775 in Java_org_apache_gluten_datasource_DatasourceJniWrapper_close ()
from /home/work/zjh/bbs/spark_tmp/gluten-7bb8d2ed-1d73-4568-b5ff-cb63bdf7b59c/jni/ebcca964-d36c-4429-a13c-b5ca68e850ed/gluten-5234449826142957659/libgluten.so
#12 0x00007f612d018767 in ?? ()
#13 0x00007f5ff178fca8 in ?? ()
#14 0x00007f612d0184c7 in ?? ()
#15 0x00007f612d018482 in ?? ()
#16 0x00007f5ff178fc68 in ?? ()
#17 0x00007f5fe36470e8 in ?? ()
#18 0x00007f5ff178fcd8 in ?? ()
#19 0x00007f5fe3647298 in ?? ()
#20 0x0000000000000000 in ?? ()**
philo-he commentedon Jun 19, 2024
@RaoZhiRou-Z, it's strange. Could you share some details about your test for reproducing?