(This is the flip side of ARROW-12960.)
Currently the Arrow compute kernel is_null always treats NaN as a non-missing value, returning false at positions of the input datum with value NaN.
It would be helpful to be able to control this behavior with an option. The option could be named nan_is_null or something similar. It would default to false, consistent with current behavior. When set to true, it should check if the input datum has a floating point data type, and if so, return true at positions where the input is NaN. If the input datum has some other type, the option should be silently ignored.
Among other things, this would enable the arrow R package to evaluate is.na() consistently with the way base R does. In base R, is.na() returns TRUE on NaN. But in the arrow R package, it returns FALSE:
is.na(c(3.14, NA, NaN))
## [1] FALSE TRUE TRUE
as.vector(is.na(Array$create(c(3.14, NA, NaN))))
## [1] FALSE TRUE FALSE
I think solving this with an option in the C++ kernel is the best solution, because I suspect there are other cases in which users might want to treat NaN as a missing value. However, it would also be possible to solve this just in the R package, by defining a mapping of is.na in the R package that checks if the input x has a floating point data type, and if so, evaluates is.na\(x\) | is.nan\(x\). If we choose to go that route, we should change this Jira issue summary to "[R] Make is.na(NaN) consistent with base R".
Reporter: Ian Cook / @ianmcook
Assignee: Christian Cordova / @Christian8491
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-12959. Please see the migration documentation for further details.
(This is the flip side of ARROW-12960.)
Currently the Arrow compute kernel
is_nullalways treatsNaNas a non-missing value, returningfalseat positions of the input datum with valueNaN.It would be helpful to be able to control this behavior with an option. The option could be named
nan_is_nullor something similar. It would default tofalse, consistent with current behavior. When set totrue, it should check if the input datum has a floating point data type, and if so, returntrueat positions where the input isNaN. If the input datum has some other type, the option should be silently ignored.Among other things, this would enable the
arrowR package to evaluateis.na()consistently with the way base R does. In base R,is.na()returnsTRUEonNaN. But in thearrowR package, it returnsFALSE:I think solving this with an option in the C++ kernel is the best solution, because I suspect there are other cases in which users might want to treat
NaNas a missing value. However, it would also be possible to solve this just in the R package, by defining a mapping ofis.nain the R package that checks if the inputxhas a floating point data type, and if so, evaluatesis.na\(x\) | is.nan\(x\). If we choose to go that route, we should change this Jira issue summary to "[R] Make is.na(NaN) consistent with base R".Reporter: Ian Cook / @ianmcook
Assignee: Christian Cordova / @Christian8491
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-12959. Please see the migration documentation for further details.