0

Each piece of this code runs great by itself on a single case...There are 450 timestamps each with +60k agents, and if I run one all alone (outside of the for loop) it finishes in about 2 seconds. Why does running them in the for loop take so long? Shouldn't it take 450*2 seconds? little.my.df has 50k rows and eligible.df has about 6300 rows.

libary(SearchTrees)
### Make a column to put my result
eligible.df$withinradius <- vector(length = dim(eligible.df)[1])

### For loop selects which rows from little.my.df are in the same
### timestamp [i,1] and are not the same agent [i,3]. 
### There are 450 timestamps.

for (i in 1:dim(eligible.df)[1]){
     timestamp.select <- little.my.df[
                                 which(
                                    little.my.df[,1] == eligible.df[i,1] & 
                                    little.my.df != eligible.df[i,3]),
                                    c(5,4)]
### Create a tree from timestamp.select and find the first NN from i

  test.tree <- createTree(timestamp.select,
                           treeType = 'quad',
                           dataType = 'point',
                           maxDepth = 1)
  test.lookup <- knnLookup(test.tree,
                           newdat = eligible.df[i,c(5,4)],
                           k = 1)
### Calculate the euclidian distance from the first NN and record it in the 
### blank column on the original dataframe.

 eligible.df[i,(dim(eligible.df))[2]] <- dist(matrix(
                                        data = c(eligible.df[i,c(5,4)],
                                        timestamp.select[test.lookup[1,1],]),
                                        ncol = 2, nrow = 2, byrow = TRUE))
       }

For each row in eligible.df I want to find the first nearest neighbor in little.my.df (50000 rows). The ACTUAL my.df has over 1 million rows so I am trying to speed this up, but I can't even get it to work with 50k rows.

2
  • It is not necessarily the case that your code will run N*2. Depending on the underlining algorithm and how memory is used in the packages, your issue could grow exponentially. It is difficult to support without a reproducible example, but you may find it useful to profile memory/time on a smaller subset of your problem to see if there are any bottlenecks, or at a minimum, how time grows. This might be a good place to start.
    – Peter_Evan
    Jun 20, 2018 at 13:26
  • @Peter_Evan thank you for that resource, I have never read about memory profiling before and will add this to my toolbox. I am sure this will help - thank you!!!
    – bstrain
    Jun 21, 2018 at 8:40

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.