[C++][R] Option for is_nan(null) to evaluate to false or true

[asfimport]

(This is the flip side of ARROW-12959.)

Currently the Arrow compute kernel is_nan always treats null as a missing value, returning null at positions of the input datum with null (missing) values.

It would be helpful to be able to control this behavior with an option. The option could be named value_for_null or something similar and it would take a nullable boolean scalar.  It would default to null, consistent with current behavior. When set to false or true, it would return false or true at positions of the input datum with null values.

Among other things, this would enable the arrow R package to evaluate is.nan() consistently with the way base R does. In base R, is.nan() returns FALSE on NA. But in the arrow R package, it returns NA:

> is.nan(c(3.14, NA, NaN))
##[1] FALSE FALSE  TRUE

as.vector(is.nan(Array$create(c(3.14, NA, NaN))))
##[1] FALSE    NA  TRUE

 I think solving this with an option in the C++ kernel is the best solution, because I suspect there are other cases in which users would want the ability to return all non-missing values in the output from is_nan without needing to call another kernel to fill the missing values in. However, it would also be possible to solve this just in the R package, by changing the mapping of is.nan in the R package. If we choose to go that route, we should change this Jira issue summary to "[R] Make is.nan(NA) consistent with base R".

Reporter: Ian Cook / @ianmcook

Related issues:

• [C++] Add option to is_nan kernel to return true on null (is duplicated by)
• [C++][R] Option for is_null(NaN) to evaluate to true (relates to)

_{Note: This issue was originally created as ARROW-12960. Please see the migration documentation for further details.}

[asfimport]

Ian Cook / @ianmcook:
@Christian8491  I'm curious to hear your idea for how to implement the options for this. It is different from ARROW-12959 because it must allow for three different possibilities:

• is_nan(null) returns null (current behavior)

• is_nan(null) returns true

• is_nan(null) returns false

  I think my suggestion to make the option a nullable boolean is actually not possible. Is that right?

[asfimport]

Eduardo Ponce / @edponce:
From an API perspective, invoking a function of the form "is_xxx(...)", I would only expect either a true/false value.

• Are there other boolean functions that return "null"?
• Are there other functions that given a "null" input, return other than "null"?

[asfimport]

Ian Cook / @ianmcook:
@edponce  I think that's a fair point, but I assume that it is important to preserve the existing behavior of is_nan as the default behavior in the future, and the existing behavior is: is_nan(null) returns null.

@bu2 @cyb70289 @pitrou what do you think, would it be OK for is_nan(null) to return either true or false (controlled by an option) but not have the option to return null?

[asfimport]

Eduardo Ponce / @edponce:
After more careful thought, I do understand and agree with "is_nan(null) = null". There are multiple ways of reasoning about it and having the option allows the client code to choose their way.

From Arrow perspective, "is_nan" shouldn't by default change the null semantics corresponding to the input data.
For example, if "null" is treated as False, then

is_null(is_nan([3.14, NA, NaN]))

would hide the fact that there was a "null" in the original input.

[asfimport]

Antoine Pitrou / @pitrou:
Well, we could add an option to control handling of nulls in the input. Is this required only for is_nan, or also for is_finite and is_inf?

[asfimport]

Ian Cook / @ianmcook:
I haven't opened Jiras for them all yet, but eventually I think we will want to add options to allow users to choose how nulls are handled by is_finite and is_inf and also how nans are handled by various other functions including coalesce. This is kind of a nuisance but it seems important because it will allow us to provide consistent, expected behavior to different bindings and front-ends.

For example, in R, is.finite(NA) returns FALSE, whereas is_finite(null) returns null in Arrow.

[asfimport]

Christian Cordova / @Christian8491:
As a humble opinion (and as a user), i do would prefer "is_nan()" always returns True or False as the base R and numpy does. If someone wants to check for the input data always can use "is_null()" and combine operations to get what they want.

[asfimport]

Antoine Pitrou / @pitrou:
What is the status on this?

[asfimport]

Dewey Dunnington / @paleolimbot:
Reprex:

library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)

record_batch(
  dbl = c(1, NA, NaN),
  float = Array$create(c(1, NA, NaN))$cast(float32())
) %>% 
  transmute(
    dbl_is_nan = is.nan(dbl),
    float_is_nan = is.nan(float)
  ) %>% 
  collect()
#> # A tibble: 3 × 2
#>   dbl_is_nan float_is_nan
#>   <lgl>      <lgl>       
#> 1 FALSE      FALSE       
#> 2 FALSE      FALSE       
#> 3 TRUE       TRUE

Where this lives in the package:

https://github.com/apache/arrow/blob/master/r/R/dplyr-functions.R#L96-L105

https://github.com/apache/arrow/blob/master/r/tests/testthat/test-dplyr-funcs-type.R#L182-L203

https://github.com/apache/arrow/blob/master/r/src/compute.cpp#L129-L130

https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/api_scalar.h#L295-L302

[asfimport]

Ian Cook / @ianmcook:
As seen in the first link Dewey shared above, the R bindings currently translate the R expression

is.nan(x)

to the Arrow expression 

and_kleene(is_nan(x), is_valid(x))

when x has a floating point data type. This is for consistency with the behavior of R.

The gist of this Jira is that we would like to simplify that translation (and other bindings' translations) so they can simply call is_nan with an option.

[asfimport]

Neal Richardson / @nealrichardson:
@Christian8491 are you working on this? If not, can you please unassign it so someone else could pick it up?

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.