Skip to content

Support remaining functions in protobuf serialization, add expr_fn for StructFunction #8100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 4 commits into from
Nov 10, 2023
Merged

Support remaining functions in protobuf serialization, add expr_fn for StructFunction #8100

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
732473f
working fix for #8098
Nov 8, 2023
ed9c041
Added enum match for StringToArray
JacobOgle Nov 9, 2023
510a120
Added enum match for StructFun as well as mapping to a supporting sca…
JacobOgle Nov 9, 2023
53ccec1
cargo fmt
JacobOgle Nov 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions datafusion/proto/src/logical_plan/from_proto.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,13 @@ use datafusion_expr::{
array_has, array_has_all, array_has_any, array_length, array_ndims, array_position,
array_positions, array_prepend, array_remove, array_remove_all, array_remove_n,
array_repeat, array_replace, array_replace_all, array_replace_n, array_slice,
array_to_string, ascii, asin, asinh, atan, atan2, atanh, bit_length, btrim,
cardinality, cbrt, ceil, character_length, chr, coalesce, concat_expr,
array_to_string, arrow_typeof, ascii, asin, asinh, atan, atan2, atanh, bit_length,
btrim, cardinality, cbrt, ceil, character_length, chr, coalesce, concat_expr,
concat_ws_expr, cos, cosh, cot, current_date, current_time, date_bin, date_part,
date_trunc, decode, degrees, digest, encode, exp,
expr::{self, InList, Sort, WindowFunction},
factorial, floor, from_unixtime, gcd, isnan, iszero, lcm, left, ln, log, log10, log2,
factorial, flatten, floor, from_unixtime, gcd, isnan, iszero, lcm, left, ln, log,
log10, log2,
logical_plan::{PlanType, StringifiedPlan},
lower, lpad, ltrim, md5, nanvl, now, nullif, octet_length, pi, power, radians,
random, regexp_match, regexp_replace, repeat, replace, reverse, right, round, rpad,
Expand Down Expand Up @@ -1645,6 +1646,13 @@ pub fn parse_expr(
)),
ScalarFunction::Isnan => Ok(isnan(parse_expr(&args[0], registry)?)),
ScalarFunction::Iszero => Ok(iszero(parse_expr(&args[0], registry)?)),
ScalarFunction::ArrowTypeof => {
Ok(arrow_typeof(parse_expr(&args[0], registry)?))
}
ScalarFunction::ToTimestamp => {
Ok(to_timestamp_seconds(parse_expr(&args[0], registry)?))
}
ScalarFunction::Flatten => Ok(flatten(parse_expr(&args[0], registry)?)),
_ => Err(proto_error(
"Protobuf deserialization error: Unsupported scalar function",
)),
Copy link
Copy Markdown
Contributor

@alamb alamb Nov 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please remove this catch all clause so the compiler will ensure all enum variants are covered?

Suggested change
_ => Err(proto_error(
"Protobuf deserialization error: Unsupported scalar function",
)),

Copy link
Copy Markdown
Contributor Author

@JacobOgle JacobOgle Nov 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a problem! I will start chipping away at the last two enum variants here in a bit. Thanks for the feedback and openness allowing the contribution! 🙏

Copy link
Copy Markdown
Contributor Author

@JacobOgle JacobOgle Nov 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alamb I was able to get the remaining cases covered with the exception to datafusion::ScalarFunction::StructFun.

Browsing the code I am not seeing a supported function for this enum variant in expr_fn.rs. The scalar_expr macro maps to enum variants in built_in_functions.rs but has no match for StructFun. Should this be mapped to the Struct variant in the macro?

Apologies for the confusion, but appreciate any guidance.

macro_rules! scalar_expr {
    ($ENUM:ident, $FUNC:ident, $($arg:ident)*, $DOC:expr) => {
        #[doc = $DOC ]
        pub fn $FUNC($($arg: Expr),*) -> Expr {
            Expr::ScalarFunction(ScalarFunction::new(
                built_in_function::BuiltinScalarFunction::$ENUM,
                vec![$($arg),*],
            ))
        }
    };
}

but only has

...
    // struct functions
    /// struct
    Struct,
...

in the BuiltinScalarFunction::$ENUM

EDIT:
To follow up, if I add
scalar_expr!(Struct, struct_fun, val, "returns a vector of fields from the struct"); to expr_fn.rs this passes the compiler checks and the tests. I just want to make sure this is the correct behavior before I commit.

Expand Down