0

I have a dataframe that includes start_date and end_date for a given unit_id along with the unit's group.

in_df <- data.frame(unit_id = c(1,2,3),
                 start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12")),
                 end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30")), 
                 group = c("pass","fail","pass"))

For each unit_id, I need to calculate the proportion of all units that pass within the duration, start_date and end_date for the current unit_id.

Taking unit_id=1 as an example, I need to find all units that have start_date and/or end_date within the dates for unit 1, i.e. start_date = 2019-01-01 and end_date = 2019-02-06. Given my in_df, this would return two units, 1 and 2. One unit passes and one fails so the proportion of pass would be 0.5. desired_df shows the output I expect for this example.

desired_df <- data.frame(unit_id = c(1,2,3),
                 start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12")),
                 end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30")), 
                 group = c("pass","fail","pass"),
                 pass_prop = c(0.5,0.5,1))

What I've tried

There are a lot of existing posts related to identifying overlapping dates. I've tried to work through some to see if I can figure this out but haven't been successful.

The following is the closest that I've gotten. It does what I want on my toy example but not on the real data (additional example data below).

library(dplyr)
library(ivs)

in_df <- data.frame(unit_id = c(1,2,3),
                 start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12")),
                 end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30")), 
                 group = c("pass","fail","pass"))


desired_df <- data.frame(unit_id = c(1,2,3),
                 start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12")),
                 end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30")), 
                 group = c("pass","fail","pass"),
                 pass_prop = c(0.5,0.5,1))

in_df <- in_df %>%
  mutate(
    start_dt = as.Date(start_date),
    end_dt = as.Date(end_date)
  ) %>%
  mutate(
    range = iv(start_dt, end_dt),
    .keep = "unused"
  )

in_df$row_n <- 1:nrow(in_df)

in_df <- in_df %>%
  group_by(group) %>%
  mutate(groupDate = iv_identify_group(range)) %>%
  group_by(groupDate, .add = TRUE)

groupCount <- in_df %>% group_by(groupDate) %>% dplyr::summarize(totalCount=n())
durationCount <- in_df %>% group_by(groupDate,group) %>% dplyr::summarize(groupCount=n())

durationCount <- dplyr::inner_join(groupCount,durationCount, by = "groupDate") 
durationCount$pass_prop <- durationCount$groupCount/durationCount$totalCount
durationCount <- filter(durationCount, group == "pass")
desired_df <- dplyr::full_join(in_df,durationCount, by = "groupDate")             
desired_df

enter image description here

The above displays exactly what I need under pass_prop. The problem with this is that iv_identify_group extends the groupDate too far when additional dates overlap as shown below.

Take unit = 1 as an example again. If I add another row to in_df that overlaps with unit = 1 and unit = 3, then the groupDate gets extended to include the ranges for units 1,2, and 4. This happens because unit 1 overlaps with 2 and 2 overlaps with 4. I want it to stop at the overlap with unit 2 since the range of unit 1 does not overlap with unit 4. Below displays this undesired output.

in_df <- data.frame(unit_id = c(1,2,3,4),
                    start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12","2019-02-20")),
                    end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30","2020-01-30")), 
                    group = c("pass","fail","pass","pass"))
# execute same code as above

enter image description here

1
  • I think a range can have the following relationships with another range: no overlap, partial overlap, exact overlap, the first encompasses the second, or the second encompasses the first. Which of these should be included?
    – Jon Spring
    Jul 15, 2022 at 5:41

2 Answers 2

0

Perhaps this?

library(dplyr)
in_df %>%
  fuzzyjoin::fuzzy_left_join(
    in_df, by = c("start_date" = "end_date", "end_date" = "start_date"),
    match_fun = list(`<=`, `>=`)) %>%
  group_by(unit_id = unit_id.x, start_date = start_date.x,
           end_date = end_date.x, group = group.x) %>%
  summarize(pass_prop = sum(group.y == "pass") / n(), .groups = "drop")

Result

  unit_id start_date end_date   group pass_prop
    <dbl> <date>     <date>     <chr>     <dbl>
1       1 2019-01-01 2019-02-06 pass        0.5
2       2 2019-02-05 2019-02-28 fail        0.5
3       3 2020-01-12 2020-01-30 pass        1 
  
0

I think ivs can help you, but I think you might be looking for iv_locate_overlaps() here:

library(ivs)
library(tidyverse)

# Starting with the more complex example with the 4th row
in_df <- tibble(unit_id = c(1,2,3,4),
                start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12","2019-02-20")),
                end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30","2020-01-30")), 
                group = c("pass","fail","pass","pass"))

in_df <- in_df %>%
  mutate(range = iv(start_date, end_date), .keep = "unused")

in_df
#> # A tibble: 4 × 3
#>   unit_id group                    range
#>     <dbl> <chr>               <iv<date>>
#> 1       1 pass  [2019-01-01, 2019-02-06)
#> 2       2 fail  [2019-02-05, 2019-02-28)
#> 3       3 pass  [2020-01-12, 2020-01-30)
#> 4       4 pass  [2019-02-20, 2020-01-30)

# "find all units that have `start_date` and/or `end_date` within the dates for unit i"
# So you are looking for "any" kind of overlap.
# `iv_locate_overlaps()` does: "For each `needle`, find every location in `haystack`
# where that `needle` has ANY overlap at all"
locs <- iv_locate_overlaps(
  needles = in_df$range, 
  haystack = in_df$range, 
  type = "any"
)

# Note `needle` 1 overlaps `haystack` locations 1 and 2 (which is what you said
# you want for unit 1)
locs
#>    needles haystack
#> 1        1        1
#> 2        1        2
#> 3        2        1
#> 4        2        2
#> 5        2        4
#> 6        3        3
#> 7        3        4
#> 8        4        2
#> 9        4        3
#> 10       4        4

# Slice `in_df` appropriately, keeping relevant columns needed to answer the question
needles <- in_df[locs$needles, c("unit_id", "range")]
haystack <- in_df[locs$haystack, c("group", "range")]
haystack <- rename(haystack, overlaps = range)

expanded_df <- bind_cols(needles, haystack)
expanded_df
#> # A tibble: 10 × 4
#>    unit_id                    range group                 overlaps
#>      <dbl>               <iv<date>> <chr>               <iv<date>>
#>  1       1 [2019-01-01, 2019-02-06) pass  [2019-01-01, 2019-02-06)
#>  2       1 [2019-01-01, 2019-02-06) fail  [2019-02-05, 2019-02-28)
#>  3       2 [2019-02-05, 2019-02-28) pass  [2019-01-01, 2019-02-06)
#>  4       2 [2019-02-05, 2019-02-28) fail  [2019-02-05, 2019-02-28)
#>  5       2 [2019-02-05, 2019-02-28) pass  [2019-02-20, 2020-01-30)
#>  6       3 [2020-01-12, 2020-01-30) pass  [2020-01-12, 2020-01-30)
#>  7       3 [2020-01-12, 2020-01-30) pass  [2019-02-20, 2020-01-30)
#>  8       4 [2019-02-20, 2020-01-30) fail  [2019-02-05, 2019-02-28)
#>  9       4 [2019-02-20, 2020-01-30) pass  [2020-01-12, 2020-01-30)
#> 10       4 [2019-02-20, 2020-01-30) pass  [2019-02-20, 2020-01-30)

# Compute the pass proportion per unit
expanded_df %>%
  group_by(unit_id) %>%
  summarise(pass_prop = sum(group == "pass") / length(group))
#> # A tibble: 4 × 2
#>   unit_id pass_prop
#>     <dbl>     <dbl>
#> 1       1     0.5  
#> 2       2     0.667
#> 3       3     1    
#> 4       4     0.667

Created on 2022-07-19 by the reprex package (v2.0.1)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.