13

I am attempting to work through Hadley Wickham's R for Data Science and have gotten tripped up on the following question: "How could you use arrange() to sort all missing values to the start? (Hint: use is.na())" I am using the flights dataset included in the nycflights13 package. Given that arrange() sorts all unknown values to the bottom of the dataframe, I am not sure how one would do the opposite across the missing values of all variables. I realize that this question can be answered with base R code, but I am specifically interested in how this would be done using dplyr and a call to the arrange() and is.na() functions. Thanks.

1
  • 7
    You can use arrange(dat, desc(is.na(variable)))
    – akrun
    Jun 11, 2016 at 6:14

4 Answers 4

11

We can wrap it with desc to get the missing values at the start

flights %>% 
    arrange(desc(is.na(dep_time)),
           desc(is.na(dep_delay)),
           desc(is.na(arr_time)), 
           desc(is.na(arr_delay)),
           desc(is.na(tailnum)),
           desc(is.na(air_time)))

The NA values were only found in those variables based on

names(flights)[colSums(is.na(flights)) >0]
#[1] "dep_time"  "dep_delay" "arr_time"  "arr_delay" "tailnum"   "air_time" 

Instead of passing each variable name at a time, we can also use NSE arrange_

nm1 <- paste0("desc(is.na(", names(flights)[colSums(is.na(flights)) >0], "))")

r1 <- flights %>%
        arrange_(.dots = nm1) 

r1 %>%
   head()
#year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay carrier flight tailnum
#  <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>     <dbl>   <chr>  <int>   <chr>
#1  2013     1     2       NA           1545        NA       NA           1910        NA      AA    133    <NA>
#2  2013     1     2       NA           1601        NA       NA           1735        NA      UA    623    <NA>
#3  2013     1     3       NA            857        NA       NA           1209        NA      UA    714    <NA>
#4  2013     1     3       NA            645        NA       NA            952        NA      UA    719    <NA>
#5  2013     1     4       NA            845        NA       NA           1015        NA      9E   3405    <NA>
#6  2013     1     4       NA           1830        NA       NA           2044        NA      9E   3716    <NA>
#Variables not shown: origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
#  time_hour <time>.

Update

With the newer versions of tidyverse (dplyr_0.7.3, rlang_0.1.2) , we can also make use of arrange_at, arrange_all, arrange_if

nm1 <- names(flights)[colSums(is.na(flights)) >0]
r2 <- flights %>% 
          arrange_at(vars(nm1), funs(desc(is.na(.))))

Or use arrange_if

f <- rlang::as_function(~ any(is.na(.)))
r3 <- flights %>% 
          arrange_if(f, funs(desc(is.na(.))))


identical(r1, r2)
#[1] TRUE

identical(r1, r3)
#[1] TRUE
5

Try the easiest way, what he just showed you:

arrange(flights, desc(is.na(dep_time)))

The other nice shortcuts:

arrange(flights, !is.na(dep_time))

or

arrange(flights, -is.na(dep_time))
3

The following arranges the rows in descending order by their number of NAs:

flights %>% 
    arrange(desc(rowSums(is.na(.))))

    # A tibble: 336,776 × 19
    year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
   <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
1   2013     1     2       NA           1545        NA       NA           1910
2   2013     1     2       NA           1601        NA       NA           1735
3   2013     1     3       NA            857        NA       NA           1209
4   2013     1     3       NA            645        NA       NA            952
5   2013     1     4       NA            845        NA       NA           1015
6   2013     1     4       NA           1830        NA       NA           2044
7   2013     1     5       NA            840        NA       NA           1001
8   2013     1     7       NA            820        NA       NA            958
9   2013     1     8       NA           1645        NA       NA           1838
10  2013     1     9       NA            755        NA       NA           1012
# ... with 336,766 more rows, and 11 more variables: arr_delay <dbl>, carrier <chr>,
#   flight <int>, tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>,
#   distance <dbl>, hour <dbl>, minute <dbl>, time_hour <dttm>
0

Solution by @akrun works fine. However, arrange_ is deprecated SE versions of main verbs. to avoid it, we can use eval

nmf <- names(flights)[colSums(is.na(flights)) > 0]
rules = paste0("!is.na(", nmf, ")")
rc <- paste(rules, collapse = ",")
arce <-  paste("arrange(flights," , rc , ")")
expr <- parse(text = arce)
ret <- eval(expr)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.