I have a dataframe that includes start_date
and end_date
for a given unit_id
along with the unit's group
.
in_df <- data.frame(unit_id = c(1,2,3),
start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12")),
end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30")),
group = c("pass","fail","pass"))
For each unit_id
, I need to calculate the proportion of all units that pass
within the duration, start_date
and end_date
for the current unit_id
.
Taking unit_id=1
as an example, I need to find all units that have start_date
and/or end_date
within the dates for unit 1
, i.e. start_date = 2019-01-01
and end_date = 2019-02-06
. Given my in_df
, this would return two units
, 1
and 2
. One unit passes and one fails so the proportion of pass would be 0.5
. desired_df
shows the output I expect for this example.
desired_df <- data.frame(unit_id = c(1,2,3),
start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12")),
end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30")),
group = c("pass","fail","pass"),
pass_prop = c(0.5,0.5,1))
What I've tried
There are a lot of existing posts related to identifying overlapping dates. I've tried to work through some to see if I can figure this out but haven't been successful.
The following is the closest that I've gotten. It does what I want on my toy example but not on the real data (additional example data below).
library(dplyr)
library(ivs)
in_df <- data.frame(unit_id = c(1,2,3),
start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12")),
end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30")),
group = c("pass","fail","pass"))
desired_df <- data.frame(unit_id = c(1,2,3),
start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12")),
end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30")),
group = c("pass","fail","pass"),
pass_prop = c(0.5,0.5,1))
in_df <- in_df %>%
mutate(
start_dt = as.Date(start_date),
end_dt = as.Date(end_date)
) %>%
mutate(
range = iv(start_dt, end_dt),
.keep = "unused"
)
in_df$row_n <- 1:nrow(in_df)
in_df <- in_df %>%
group_by(group) %>%
mutate(groupDate = iv_identify_group(range)) %>%
group_by(groupDate, .add = TRUE)
groupCount <- in_df %>% group_by(groupDate) %>% dplyr::summarize(totalCount=n())
durationCount <- in_df %>% group_by(groupDate,group) %>% dplyr::summarize(groupCount=n())
durationCount <- dplyr::inner_join(groupCount,durationCount, by = "groupDate")
durationCount$pass_prop <- durationCount$groupCount/durationCount$totalCount
durationCount <- filter(durationCount, group == "pass")
desired_df <- dplyr::full_join(in_df,durationCount, by = "groupDate")
desired_df
The above displays exactly what I need under pass_prop
. The problem with this is that iv_identify_group
extends the groupDate
too far when additional dates overlap as shown below.
Take unit = 1
as an example again. If I add another row to in_df
that overlaps with unit = 1
and unit = 3
, then the groupDate
gets extended to include the ranges for units
1
,2
, and 4
. This happens because unit 1
overlaps with 2
and 2
overlaps with 4
. I want it to stop at the overlap with unit 2
since the range of unit 1
does not overlap with unit 4
. Below displays this undesired output.
in_df <- data.frame(unit_id = c(1,2,3,4),
start_date = as.Date(c("2019-01-01","2019-02-05","2020-01-12","2019-02-20")),
end_date = as.Date(c("2019-02-06","2019-02-28","2020-01-30","2020-01-30")),
group = c("pass","fail","pass","pass"))
# execute same code as above