[GLUTEN-1589][CH] support udf for clickhouse backend by taiyang-li · Pull Request #1596 · apache/gluten

taiyang-li · 2023-05-09T11:03:12Z

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

(Fixes: #1589)

除了支持udf之外，还解决了clickhouse中gtest和benchmark的编译失败问题。

github-actions · 2023-05-09T11:03:31Z

#1589

github-actions · 2023-05-09T11:03:33Z

Run Gluten Clickhouse CI

github-actions · 2023-05-09T14:41:36Z

Run Gluten Clickhouse CI

github-actions · 2023-05-10T03:32:21Z

Run Gluten Clickhouse CI

github-actions · 2023-05-10T04:09:09Z

Run Gluten Clickhouse CI

taiyang-li · 2023-05-10T04:26:17Z

test hive udf

0: jdbc:hive2://localhost:10000/> CREATE temporary FUNCTION indigouidhash AS 'sg.bigo.hive.udf.IndigoUidHash' ;
+---------+
| Result  |
+---------+

+---------+
No rows selected (0.072 seconds)

0: jdbc:hive2://localhost:10000/> explain formatted select indigouidhash(cast(id as String)) from range(10);  

+----------------------------------------------------+
|                        plan                        |
+----------------------------------------------------+
| == Physical Plan ==
CHNativeColumnarToRow (5)
+- ^ ProjectExecTransformer (3)
   +- RowToCHNativeColumnar (2)
      +- * Range (1)


(1) Range [codegen id : 1]
Output [1]: [id#13L]
Arguments: Range (0, 10, step=1, splits=None)

(2) RowToCHNativeColumnar
Input [1]: [id#13L]

(3) ProjectExecTransformer
Input [1]: [id#13L]
Arguments: [HiveSimpleUDF#sg.bigo.hive.udf.IndigoUidHash(cast(id#13L as string)) AS indigouidhash(CAST(id AS STRING))#14]

(4) WholeStageCodegenTransformer (1)
Input [1]: [indigouidhash(CAST(id AS STRING))#14]

(5) CHNativeColumnarToRow
Input [1]: [indigouidhash(CAST(id AS STRING))#14]

 |
+----------------------------------------------------+

github-actions · 2023-05-10T04:28:00Z

Run Gluten Clickhouse CI

github-actions · 2023-05-10T04:30:20Z

Run Gluten Clickhouse CI

taiyang-li · 2023-05-10T10:16:14Z

test python udf:

#!/usr/bin/python3.7
import time

from pyspark import SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.types import LongType

conf = SparkConf()
spark = SparkSession.builder.config(conf=conf).getOrCreate()
spark.sparkContext.setLogLevel("DEBUG")
# sc = SparkContext(conf=conf)
# spark = SparkSession.builder.master("local[1]").appName("test_udf").getOrCreate()


def my_add(x, y):
    return x + y


def print_nothing(x):
    pass


spark.udf.register("my_add", my_add, LongType())
start_time = time.time()

print(
    spark.sql(
        # "explain formatted select cast(id as String) from range(100)"
	"explain formatted select my_add(id, 1) from range(1000000)"
    ).collect()
)

end_time = time.time()
time_cost = end_time - start_time
print("Time cost: ", time_cost)

== Physical Plan ==
CHNativeColumnarToRow (6)
+- ^ ProjectExecTransformer (4)
   +- ^ EvalPythonExecTransformer (3)
      +- RowToCHNativeColumnar (2)
         +- * Range (1)


(1) Range [codegen id : 1]
Output [1]: [id#5L]
Arguments: Range (0, 1000000, step=1, splits=None)

(2) RowToCHNativeColumnar
Input [1]: [id#5L]

(3) EvalPythonExecTransformer
Input [1]: [id#5L]
Arguments: [my_add(id#5L, 1)], [pythonUDF0#8L]

(4) ProjectExecTransformer
Input [2]: [id#5L, pythonUDF0#8L]
Arguments: [pythonUDF0#8L AS my_add(id, 1)#7L]

(5) WholeStageCodegenTransformer (1)
Input [1]: [my_add(id, 1)#7L]

(6) CHNativeColumnarToRow
Input [1]: [my_add(id, 1)#7L]

github-actions · 2023-05-10T12:12:45Z

Run Gluten Clickhouse CI

taiyang-li · 2023-05-10T12:14:11Z

@philo-he @zzcclp do you have time to review this pr, thanks

github-actions · 2023-05-11T08:06:06Z

Run Gluten Clickhouse CI

philo-he

Just left some comments for common scala code part. Thanks!

github-actions · 2023-05-11T10:10:16Z

Run Gluten Clickhouse CI

github-actions · 2023-05-11T10:44:32Z

Run Gluten Clickhouse CI

github-actions · 2023-05-12T06:52:34Z

Run Gluten Clickhouse CI

github-actions · 2023-05-15T03:19:52Z

Run Gluten Clickhouse CI

github-actions · 2023-05-15T03:23:46Z

Run Gluten Clickhouse CI

github-actions · 2023-05-15T03:27:34Z

Run Gluten Clickhouse CI

github-actions · 2023-05-15T03:57:16Z

Run Gluten Clickhouse CI

github-actions · 2023-05-15T08:47:49Z

Run Gluten Clickhouse CI

github-actions · 2023-05-15T08:52:09Z

Run Gluten Clickhouse CI

github-actions · 2023-05-24T09:50:33Z

Run Gluten Clickhouse CI

taiyang-li · 2023-05-25T02:26:08Z

@rui-mo velox failed with the same logs which appeared in other prs previously. Do you have time to take a look.

2023-05-24T12:56:32.6577333Z Failed queries (a failed query with correct row count indicates value mismatches): 
2023-05-24T12:56:32.6577605Z 
2023-05-24T12:56:32.6577957Z |       Query ID|     Was Passed|            Expected Row Count|              Actual Row Count|  Baseline Query Time (Millis)|           Query Time (Millis)|          Query Time Variation|
2023-05-24T12:56:32.6578507Z |            q95|          false|                           N/A|                           N/A|                           N/A|                           N/A|                           N/A|

github-actions · 2023-05-25T03:18:19Z

Run Gluten Clickhouse CI

rui-mo · 2023-05-25T05:30:52Z

@taiyang-li Looks not relevant to this PR. Do we need to increase the memory limit? cc @zhztheplayer

2023-05-25T04:53:20.8886192Z Executing SQL query from resource path /tpcds-queries/q95.sql...
2023-05-25T04:53:27.3620707Z 23/05/25 12:53:27 ERROR TaskResources: Task 107 failed by error: 
2023-05-25T04:53:27.3622556Z org.apache.spark.SparkException: Can't acquire 67108864 bytes memory to build hash relation, got 56478455 bytes
2023-05-25T04:53:27.3624615Z 	at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotAcquireMemoryToBuildLongHashedRelationError(QueryExecutionErrors.scala:795)
2023-05-25T04:53:27.3632966Z 	at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.ensureAcquireMemory(HashedRelation.scala:581)
2023-05-25T04:53:27.3649896Z 	at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.grow(HashedRelation.scala:813)
2023-05-25T04:53:27.3650603Z 	at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.append(HashedRelation.scala:761)
2023-05-25T04:53:27.3651254Z 	at org.apache.spark.sql.execution.joins.LongHashedRelation$.apply(HashedRelation.scala:1064)
2023-05-25T04:53:27.3730412Z 	at org.apache.spark.sql.execution.joins.HashedRelation$.apply(HashedRelation.scala:153)

github-actions · 2023-05-25T07:03:17Z

Run Gluten Clickhouse CI

github-actions · 2023-05-26T02:10:48Z

Run Gluten Clickhouse CI

github-actions · 2023-05-29T02:41:52Z

Run Gluten Clickhouse CI

github-actions · 2023-05-29T02:50:50Z

Run Gluten Clickhouse CI

github-actions · 2023-05-29T11:23:43Z

Run Gluten Clickhouse CI

github-actions · 2023-05-29T12:15:53Z

Run Gluten Clickhouse CI

zzcclp · 2023-05-31T02:26:43Z

please rebase

github-actions · 2023-06-06T02:52:32Z

Run Gluten Clickhouse CI

github-actions · 2023-06-06T10:08:15Z

Run Gluten Clickhouse CI

github-actions · 2023-06-06T11:52:20Z

Run Gluten Clickhouse CI

baibaichen · 2023-06-07T02:32:29Z

LGTM

github-actions · 2023-06-07T06:56:01Z

Run Gluten Clickhouse CI

…1596)" This reverts commit 5e75698.

taiyang-li marked this pull request as draft May 9, 2023 11:03

taiyang-li force-pushed the gluten_1589 branch from ae5c3cc to d89f30d Compare May 10, 2023 03:31

philo-he reviewed May 11, 2023

View reviewed changes

taiyang-li force-pushed the gluten_1589 branch from d6ae7bf to eac2d07 Compare May 15, 2023 02:53

taiyang-li force-pushed the gluten_1589 branch from e83b7b1 to 9529110 Compare May 15, 2023 03:23

taiyang-li marked this pull request as ready for review May 15, 2023 08:51

taiyang-li force-pushed the gluten_1589 branch from cda469c to 2ea74ce Compare May 15, 2023 08:51

baibaichen reviewed May 15, 2023

View reviewed changes

Comment thread cpp-ch/local-engine/Functions/my_udf/myAdd.cpp Outdated

baibaichen reviewed May 15, 2023

View reviewed changes

Comment thread ...-clickhouse/src/test/scala/io/glutenproject/execution/GlutenClickHouseTPCHParquetSuite.scala Outdated

taiyang-li force-pushed the gluten_1589 branch from ca7e7f7 to 6ce0d83 Compare May 24, 2023 09:50

taiyang-li force-pushed the gluten_1589 branch from bb325f5 to ec5393f Compare May 29, 2023 02:41

taiyang-li force-pushed the gluten_1589 branch from 745d22a to 29851ab Compare June 6, 2023 02:52

baibaichen previously approved these changes Jun 7, 2023

View reviewed changes

taiyang-li added 3 commits June 7, 2023 14:54

support hive/python/scala udf for gluten

50861ba

fix build error

51da75b

fix failed uts

8739c79

taiyang-li dismissed baibaichen’s stale review via 8739c79 June 7, 2023 06:55

taiyang-li force-pushed the gluten_1589 branch from bbcd9a7 to 8739c79 Compare June 7, 2023 06:55

baibaichen approved these changes Jun 8, 2023

View reviewed changes

baibaichen merged commit 5e75698 into apache:main Jun 8, 2023

rui-mo added a commit to rui-mo/gazelle-jni that referenced this pull request Jun 8, 2023

Revert "[GLUTEN-1589][CH] support udf for clickhouse backend (apache#…

12a009a

…1596)" This reverts commit 5e75698.

Conversation

taiyang-li commented May 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Uh oh!

github-actions Bot commented May 9, 2023

Uh oh!

github-actions Bot commented May 9, 2023

Uh oh!

github-actions Bot commented May 9, 2023

Uh oh!

github-actions Bot commented May 10, 2023

Uh oh!

github-actions Bot commented May 10, 2023

Uh oh!

taiyang-li commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 10, 2023

Uh oh!

github-actions Bot commented May 10, 2023

Uh oh!

taiyang-li commented May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 10, 2023

Uh oh!

taiyang-li commented May 10, 2023

Uh oh!

github-actions Bot commented May 11, 2023

Uh oh!

philo-he left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 11, 2023

Uh oh!

github-actions Bot commented May 11, 2023

Uh oh!

github-actions Bot commented May 12, 2023

Uh oh!

github-actions Bot commented May 15, 2023

Uh oh!

github-actions Bot commented May 15, 2023

Uh oh!

github-actions Bot commented May 15, 2023

Uh oh!

github-actions Bot commented May 15, 2023

Uh oh!

github-actions Bot commented May 15, 2023

Uh oh!

github-actions Bot commented May 15, 2023

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 24, 2023

Uh oh!

taiyang-li commented May 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 25, 2023

Uh oh!

rui-mo commented May 25, 2023

Uh oh!

github-actions Bot commented May 25, 2023

Uh oh!

github-actions Bot commented May 26, 2023

Uh oh!

github-actions Bot commented May 29, 2023

Uh oh!

github-actions Bot commented May 29, 2023

Uh oh!

taiyang-li commented May 9, 2023 •

edited

Loading

taiyang-li commented May 10, 2023 •

edited

Loading

taiyang-li commented May 10, 2023 •

edited

Loading

philo-he left a comment •

edited

Loading

taiyang-li commented May 25, 2023 •

edited

Loading