2

I have received a script that essentially attaches two data sources together based on a time-stamp. It takes very long to run. So I figured to rewrite it. So let me shortly explain the problem:

THE DATA

The data that comes from two different sources. Let's call them source A and B to keep things simple. Source A records for example concentration of CO2 of the environment twice every second. Source B contains observations of individual cows during a milking in a robot. So the robot records milk yield, for example, from start till end of milking (that's one row in the table).

I quickly wrote a simplified representation of what the data looks like:

ori = "2015-03-02"
TimeStamp <- seq(from=20000,to=20025, by = 0.5)
TimeStamp <- as.POSIXct(TimeStamp[-length(TimeStamp)], origin = ori, tz ="GMT")
set.seed(42)
x1 <- 1000:2000
x1 <- sample(50)
sourceA <- data.frame(TimeStamp = TimeStamp, x1 = x1) 

StartTime <- seq(from=20000,to=20025, by = 5)
EndTime <- StartTime + 2
StartTime <- as.POSIXct(StartTime[-length(StartTime)], origin = ori, tz ="GMT")
EndTime <- as.POSIXct(EndTime[-length(EndTime)], origin = ori, tz ="GMT")
set.seed(36)
x2 <- 600:800
x2 <- sample(x2,size = 5)
sourceB <- data.frame(StartTime = StartTime, EndTime = EndTime, x2 = x2)

Originally, I wrote a script that would calculate the time difference between the TimeStamp and the StartTime. Then where the difference is 0 attach the data from source B to source A. I want to avoid looping through the data one-by-one - this will take forever.

What would be a good way to approach this?

[EDIT]

I added code to create the desired output:

tmp <- NULL
for(i in 1:5){
  tmp <- c(tmp, rep(sourceB$x2[i],6))
  tmp <- c(tmp, rep(0,4))
}
EndResult <- data.frame(TimeStamp = sourceA$TimeStamp, x1 = sourceA$x1, x2 = tmp)
12
  • 1
    your script doesn't run in R: missing "ori". Please try copy/pasting your whole code in an R console without history (on linux use R --vanilla) before posting. Also merging two different datasets on different time scales can have multiple solutions. You need to give a little more context on what you want to achieve. Typically, what would a line correspond to in your final table?
    – cmbarbu
    Mar 2, 2015 at 10:59
  • What is the desired output? The solution to this problem seems quite simple, but there is just too much text here and its not entirely clear what are you trying to achieve. Also, sourceA is just too big (1K rows). We don't need such a big data set in order to reproduce your problem. Please rescale your problem to 20-30 rows and provide how would your expected output should look like in the end. Mar 2, 2015 at 11:17
  • 1
    All you need to do is to explain what are you trying to achieve by providing your desired output (on a much smaller data set). This could illustrate your problem way better than this whole text (usually that amount of text scares people). Mar 2, 2015 at 11:22
  • 2
    @DavidArenburg I updated my question based on your helpful comments and added code for the desired output. I hope its a bit more clear. Cheers
    – tstev
    Mar 2, 2015 at 11:53
  • 1
    Don't use format, just leave them POSIXct class. My solution works perfectly with this class. Mar 2, 2015 at 12:50

1 Answer 1

3

I will assume we are dealing with real POSIXct classes here (instead of factor, like you assumingly by accident generated), then the solution is simply using the foverlaps function from the data.table package which is very efficient for such a task.

First, we will convert both data sets to data.table objects, create corresponding start and end dates in both and key the smaller data set. Then, all is left to do is to run the foverlaps function. Here are the results of the first 10 lines

library(data.table)
setkey(setDT(sourceB), StartTime, EndTime)
setnames(setDT(sourceA)[, EndTime := TimeStamp], "TimeStamp", "StartTime")
(EndResult <- foverlaps(sourceA, sourceB))
#               StartTime             EndTime  x2         i.StartTime x1           i.EndTime
#  1: 2015-03-02 05:33:20 2015-03-02 05:33:22 725 2015-03-02 05:33:20  1 2015-03-02 05:33:20
#  2: 2015-03-02 05:33:20 2015-03-02 05:33:22 725 2015-03-02 05:33:20 47 2015-03-02 05:33:20
#  3: 2015-03-02 05:33:20 2015-03-02 05:33:22 725 2015-03-02 05:33:21 21 2015-03-02 05:33:21
#  4: 2015-03-02 05:33:20 2015-03-02 05:33:22 725 2015-03-02 05:33:21  5 2015-03-02 05:33:21
#  5: 2015-03-02 05:33:20 2015-03-02 05:33:22 725 2015-03-02 05:33:22 17 2015-03-02 05:33:22
#  6:                <NA>                <NA>  NA 2015-03-02 05:33:22 13 2015-03-02 05:33:22
#  7:                <NA>                <NA>  NA 2015-03-02 05:33:23 27 2015-03-02 05:33:23
#  8:                <NA>                <NA>  NA 2015-03-02 05:33:23 23 2015-03-02 05:33:23
#  9:                <NA>                <NA>  NA 2015-03-02 05:33:24  3 2015-03-02 05:33:24
# 10:                <NA>                <NA>  NA 2015-03-02 05:33:24  4 2015-03-02 05:33:24
....
19
  • 1
    I need to generate end date in both data sets, so I just assigned the same date to end and start dates within sourceA. Did you try to run this on real data? This should be very fast. Mar 2, 2015 at 13:28
  • 1
    All you have to do is to generate end date in sourceA (which is the same as the TimeStamp. Then make sure that both data sets have same names for the joining fields (this is the setnames part in the third line), lastly, you need to key (by using setkey) sourceB` by start and end date, and run foverlaps when sourceA is the first argument and sourceB is the second. Mar 2, 2015 at 13:38
  • 1
    They should be the same because you don't want to match by intervals in sourceA, rather by row, while the intervals are sourceB. Mar 3, 2015 at 10:17
  • 1
    @DavidArenburg -- it works, and gives the desired output! Thanks a lot for being patient with my initally bad question!
    – tstev
    Mar 3, 2015 at 11:19
  • 1
    Try installing the development version from Github. It maybe a bug in the older version github.com/Rdatatable/data.table/wiki/Installation Mar 18, 2015 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.