2

In SQLite I would like to find the standard deviation of the first differences of a (logged) series that I define with GROUP BY. My data provider gives me a daily price series, but I would like to find annualized daily volatility (the standard deviation of daily returns -- first difference of the natural log of the series -- over each year). I can bring the data to R, then use ddply(), but I would like to do this entirely in SQLite. I tried the difference() function from the RSQLite.extfunctions package, but my usage is wrong. I expected it to work like diff() in R, but I can't find much documentation.

This generates some data.

stocks <- 5
years <- 5
list.n <- as.list(rep(252, stocks * years))
list.mean <- as.list(rep(0, stocks * years))
list.sd <- as.list(abs(runif(stocks * years, min = 0, max = 0.1)))
list.po <- as.list(runif(n = stocks, min = 25, max = 100)) 
list.ret <- mapply(rnorm, n = list.n, mean = list.mean, sd = list.sd, SIMPLIFY = F)
my.price <- function(po, ret) po * exp(cumsum(ret))
list.price <- mapply(my.price, po = list.po, ret = list.ret, SIMPLIFY = F)
gvkey <- rep(seq(stocks), each = 252 * years)
day <- rep(seq(252), n = stocks * years)
fyr <- rep(seq(years), n = stocks, each = 252)
data.dly <- data.frame(gvkey, fyr, day, p = unlist(list.price))

Here is how I would do it with ddply() and the result.

# I could do this easily with ddply and subset
library(plyr)
data.dly <- ddply(data.dly, .(gvkey, fyr), transform, vol = sd(diff(log(p))))
data.ann <- subset(data.dly, day == 252)
head(data.ann)
     gvkey fyr day         p         vol
252      1   1 252  86.08568 0.077287182
504      1   2 252  43.32113 0.066741862
756      1   3 252  68.69734 0.084419564
1008     1   4 252  75.37267 0.006003969
1260     1   5 252  17.53583 0.083688727
1512     2   1 252 168.44656 0.035959492

And here is my (failed) SQLite attempt and error.

# but I can't figure it out in SQLite
library(RSQLite)
library(RSQLite.extfuns)
db <- dbConnect(SQLite())
init_extensions(db)
[1] TRUE
dbWriteTable(db, name = "data_dly", value = data.dly)
[1] TRUE
temp <- dbGetQuery(db, "SELECT stdev(difference(log(p))) FROM data_dly GROUP BY gvkey, fyr ORDER BY gvkey, fyr, day")
Error in sqliteExecStatement(con, statement, bind.data) : 
  RS-DBI driver: (error in statement: wrong number of arguments to function difference())

Does difference() need a comma separated list of numbers? Can I do this entirely in SQLite? Or do I need to perform in R? Thanks!

  • 1
    Minor style point: including the prompts in your code chunks makes it a pain to copy and paste. – Richie Cotton Jul 14 '11 at 15:26
2

Try this where data.dly is the data frame in the post:

library(sqldf)
out <- sqldf("select A.gvkey, A.fyr, stdev(log(A.p) - log(B.p)) vol
    from `data.dly` A join `data.dly` B 
    where A.day = B.day + 1 
        and A.gvkey = B.gvkey 
        and A.fyr = B.fyr 
    group by A.gvkey, A.fyr")

This gives:

> head(out)
  gvkey fyr        vol
1     1   1 0.09312510
2     1   2 0.01905447
3     1   3 0.01651095
4     1   4 0.06962667
5     1   5 0.05243940
6     2   1 0.03039751
| improve this answer | |
2

The difference SQL command takes two character arguments, and has a different meaning to R's diff command.

Retrieve the data with an SQL command, then do stats using R.

temp <- dbGetQuery(db, "SELECT p FROM data_dly GROUP BY gvkey, fyr ORDER BY gvkey, fyr, day")
sd(diff(log(temp$p)))
| improve this answer | |
  • Thanks. AFAIK, there's no way to call an R function within SQLite. Is that correct? – Richard Herron Jul 14 '11 at 15:36
  • 1
    @richardh - I think the only way to do that would be to write a C function that in turn calls R, as mentioned here. Also, you might consider sending a note to the RSQLite.extfuns maintainer that perhaps difference shouldn't be listed among the math functions to avoid further confusion... – joran Jul 14 '11 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.