1

I am attempting to create new variables using a function and lapply rather than working right in the data with loops. I used to use Stata and would have solved this problem with a method similar to that discussed here.

Since naming variables programmatically is so difficult or at least awkward in R (and it seems you can't use indexing with assign), I have left the naming process until after the lapply. I am then using a for loop to do the renaming prior to merging and again for the merging. Are there more efficient ways of doing this? How would I replace the loops? Should I be doing some sort of reshaping?

#Reproducible data
data <- data.frame("custID" = c(1:10, 1:20),
    "v1" = rep(c("A", "B"), c(10,20)), 
    "v2" = c(30:21, 20:19, 1:3, 20:6), stringsAsFactors = TRUE)

#Function to analyze customer distribution for each category (v1)
pf <- function(cat, df) {

        df <- df[df$v1 == cat,]
        df <- df[order(-df$v2),]

    #Divide the customers into top percents
    nr <- nrow(df)
    p10 <- round(nr * .10, 0)
    cat("Number of people in the Top 10% :", p10, "\n")
    p20 <- round(nr * .20, 0)
    p11_20 <- p20-p10
    cat("Number of people in the 11-20% :", p11_20, "\n")

    #Keep only those customers in the top groups
    df <- df[1:p20,]

    #Create a variable to identify the percent group the customer is in
    top_pct <- integer(length = p10 + p11_20)

    #Identify those in each group
    top_pct[1:p10] <- 10
    top_pct[(p10+1):p20] <- 20

    #Add this variable to the data frame
    df$top_pct <- top_pct

    #Keep only custID and the new variable
    df <- subset(df, select = c(custID, top_pct))

    return(df)

}


##Run the customer distribution function
v1Levels <- levels(data$v1)
res <- lapply(v1Levels, pf, df = data)

#Explore the results
summary(res)

    #      Length Class      Mode
    # [1,] 2      data.frame list
    # [2,] 2      data.frame list

print(res)

    # [[1]]
    #   custID top_pct
    # 1      1      10
    # 2      2      20
    # 
    # [[2]]
    #    custID top_pct
    # 11      1      10
    # 16      6      10
    # 12      2      20
    # 17      7      20



##Merge the two data frames but with top_pct as a different variable for each category

#Change the new variable name
for(i in 1:length(res)) {
    names(res[[i]])[2] <- paste0(v1Levels[i], "_top_pct")
}

#Merge the results
res_m <- res[[1]]
for(i in 2:length(res)) {
    res_m <- merge(res_m, res[[i]], by = "custID", all = TRUE)
}

print(res_m)

    #   custID A_top_pct B_top_pct
    # 1      1        10        10
    # 2      2        20        20
    # 3      6        NA        10
    # 4      7        NA        20
8
  • 1
    You can get rid of most of this with quantile() and hist(breaks = define_your_nonuniform_breaks...)
    – smci
    May 1, 2015 at 3:35
  • Don't call your df data because that shadows a builtin.
    – smci
    May 1, 2015 at 3:36
  • 3
    When you do df <- df[df$v1 == cat,] for each categorical level, you are just doing a kludgy Split-Apply-Combine. See documentation on plyr/dplyr group_by. Try to read about R paradigms for doing common operations.
    – smci
    May 1, 2015 at 3:40
  • I retitled this "Find top-10% and 10-20% decile entries from dataframe, grouped by v1" since that's the intent of the code. You might like to rephrase the question.
    – smci
    May 1, 2015 at 4:05
  • 1
    @smci Looks like "dataframes" is not the right tag for R "data.frame"s
    – Frank
    May 1, 2015 at 4:24

3 Answers 3

7

Stick to your Stata instincts and use a single data set:

require(data.table)
DT <- data.table(data)

DT[,r:=rank(v2)/.N,by=v1]

You can see the result by typing DT.


From here, you can group the within-v1 rank, r, if you want to. Following Stata idioms...

DT[,g:={
  x = rep(0,.N)
  x[r>.8] = 20
  x[r>.9] = 10
  x
}]

This is like gen and then two replace ... if statements. Again, you can see the result with DT.


Finally, you can subset with

DT[g>0]

which gives

   custID v1 v2     r  g
1:      1  A 30 1.000 10
2:      2  A 29 0.900 20
3:      1  B 20 0.975 10
4:      2  B 19 0.875 20
5:      6  B 20 0.975 10
6:      7  B 19 0.875 20

These steps can also be chained together:

DT[,r:=rank(v2)/.N,by=v1][,g:={x = rep(0,.N);x[r>.8] = 20;x[r>.9] = 10;x}][g>0]

(Thanks to @ExperimenteR:)

To rearrange for the desired output in the OP, with values of v1 in columns, use dcast:

dcast(
  DT[,r:=rank(v2)/.N,by=v1][,g:={x = rep(0,.N);x[r>.8] = 20;x[r>.9] = 10;x}][g>0], 
  custID~v1)

Currently, dcast requires the latest version of data.table, available (I think) from Github.

2
  • 2
    dcast(DT[,r:=rank(v2)/.N,by=v1][,g:={x = rep(0,.N);x[r>.8] = 20;x[r>.9] = 10;x}][g>0], custID~v1) to put the columns side-by-side. May 1, 2015 at 4:10
  • @ExperimenteR Cool, thanks! I haven't learned/installed dcast yet.
    – Frank
    May 1, 2015 at 4:19
5

You don't need the function pf to achieve what you want. Try dplyr/tidyr combo

library(dplyr)
library(tidyr)
data %>% 
    group_by(v1) %>% 
    arrange(desc(v2))%>%
    mutate(n=n()) %>% 
    filter(row_number() <= round(n * .2)) %>% 
    mutate(top_pct= ifelse(row_number()<=round(n* .1), 10, 20)) %>%
    select(custID, top_pct) %>% 
    spread(v1,  top_pct)
#  custID  A  B
#1      1 10 10
#2      2 20 20
#3      6 NA 10
#4      7 NA 20
2
  • 1
    This is great, you beat me to it. @KevinM, read up the paradigms for these dplyr commands. (data.table is also a possible package, but much less friendly to an R beginner. So learn with dplyr.)
    – smci
    May 1, 2015 at 3:53
  • Apologies for strongly worded. Toned down as suggested. May 1, 2015 at 4:07
3

The idiomatic way to do this kind of thing in R would be to use a combination of split and lapply. You're halfway there with your use of lapply; you just need to use split as well.

lapply(split(data, data$v1), function(df) {
    cutoff <- quantile(df$v2, c(0.8, 0.9))
    top_pct <- ifelse(df$v2 > cutoff[2], 10, ifelse(df$v2 > cutoff[1], 20, NA))
    na.omit(data.frame(id=df$custID, top_pct))
})

Finding quantiles is done with quantile.

1
  • Please see my new post regarding how to combine these results. Anything you would add?
    – Kevin M
    May 1, 2015 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.