2

I ran into some unexpected behavior when filtering an SQLite database with dplyr in R. Here is an example of what I see:

library(dplyr)

# Some test data
df <- tbl_df(data.frame(
  v1 = c('a', 'b', 'a', 'b'),
  v2 = c('b', 'a', 'a', 'b'),
  v3 = month.abb[1:4]))

db <- copy_to(src_sqlite('example.sqlite', create = TRUE), df)

filter(df, v1 == 'a' | v2 == 'a', v3 == 'Jan') 
# Source: local data frame [1 x 3]
#
#   v1 v2  v3
# 1  a  b Jan

filter(db, v1 == 'a' | v2 == 'a', v3 == 'Jan')
# Source: sqlite 3.8.6 [example.sqlite]
# From: df [2 x 3]
# Filter: v1 == "a" | v2 == "a", v3 == "Jan" 
#
#  v1 v2  v3
# 1  a  b Jan
# 2  a  a Mar

I am very new to working with databases, so I feel like I'm missing something obvious here having to do with using | in a filter since this works as expected:

filter(db, v1 == 'a', v3 == 'Jan') 
# Source: sqlite 3.8.6 [example.sqlite]
# From: df [1 x 3]
# Filter: v1 == "a", v3 == "Jan" 
#
#   v1 v2  v3
# 1  a  b Jan

Should | be avoided when working in databases with dplyr?

Update: dplyr issue #934

0

1 Answer 1

4

If you do

filter(db, v1 == 'a' | v2 == 'a', v3 == 'Jan') %>% show_query()

you'll see that becomes

<SQL>
SELECT "v1", "v2", "v3"
FROM "df"
WHERE "v1" = 'a' OR "v2" = 'a' AND "v3" = 'Jan'

So the | is translated to an OR but that first "term" isn't kept together. Then since AND has a higher precedence that OR, it would be the same as running

WHERE "v1" = 'a' OR ("v2" = 'a' AND "v3" = 'Jan')

You can add your own parenthesis to avoid the problem

filter(db, (v1 == 'a' | v2 == 'a'), v3 == 'Jan') %>% show_query()
# <SQL>
# SELECT "v1", "v2", "v3"
# FROM "df"
# WHERE ("v1" = 'a' OR "v2" = 'a') AND "v3" = 'Jan'

filter(db, (v1 == 'a' | v2 == 'a'), v3 == 'Jan')
# Source: sqlite 3.8.6 [example.sqlite]
# From: df [1 x 3]
# Filter: (v1 == "a" | v2 == "a"), v3 == "Jan" 
# 
#   v1 v2  v3
# 1  a  b Jan

I'm not sure why dplyr doesn't implicitly add the parenthesis for you if you have multiple conditions in the case of SQL because it does seem to be evaluating them properly for the data.frame. You might consider filing this as an "issue" to see what the official stance on this is because to me it looks like a bug.

1
  • 1
    This is a definitely a bug
    – hadley
    Jan 29, 2015 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.