149

This is a bit of a philosophical question about data.table join syntax. I am finding more and more uses for data.tables, but still learning...

The join format X[Y] for data.tables is very concise, handy and efficient, but as far as I can tell, it only supports inner joins and right outer joins. To get a left or full outer join, I need to use merge:

  • X[Y, nomatch = NA] -- all rows in Y -- right outer join (default)
  • X[Y, nomatch = 0] -- only rows with matches in both X and Y -- inner join
  • merge(X, Y, all = TRUE) -- all rows from both X and Y -- full outer join
  • merge(X, Y, all.x = TRUE) -- all rows in X -- left outer join

It seems to me that it would be handy if the X[Y] join format supported all 4 types of joins. Is there a reason only two types of joins are supported?

For me, the nomatch = 0 and nomatch = NA parameter values are not very intuitive for the actions being performed. It is easier for me to understand and remember the merge syntax: all = TRUE, all.x = TRUE and all.y = TRUE. Since the X[Y] operation resembles merge much more than match, why not use the merge syntax for joins rather than the match function's nomatch parameter?

Here are code examples of the 4 join types:

# sample X and Y data.tables
library(data.table)
X <- data.table(t = 1:4, a = (1:4)^2)
setkey(X, t)
X
#    t  a
# 1: 1  1
# 2: 2  4
# 3: 3  9
# 4: 4 16

Y <- data.table(t = 3:6, b = (3:6)^2)
setkey(Y, t)
Y
#    t  b
# 1: 3  9
# 2: 4 16
# 3: 5 25
# 4: 6 36

# all rows from Y - right outer join
X[Y]  # default
#  t  a  b
# 1: 3  9  9
# 2: 4 16 16
# 3: 5 NA 25
# 4: 6 NA 36

X[Y, nomatch = NA]  # same as above
#    t  a  b
# 1: 3  9  9
# 2: 4 16 16
# 3: 5 NA 25
# 4: 6 NA 36

merge(X, Y, by = "t", all.y = TRUE)  # same as above
#    t  a  b
# 1: 3  9  9
# 2: 4 16 16
# 3: 5 NA 25
# 4: 6 NA 36

identical(X[Y], merge(X, Y, by = "t", all.y = TRUE))
# [1] TRUE

# only rows in both X and Y - inner join
X[Y, nomatch = 0]  
#    t  a  b
# 1: 3  9  9
# 2: 4 16 16

merge(X, Y, by = "t")  # same as above
#    t  a  b
# 1: 3  9  9
# 2: 4 16 16

merge(X, Y, by = "t", all = FALSE)  # same as above
#    t  a  b
# 1: 3  9  9
# 2: 4 16 16

identical( X[Y, nomatch = 0], merge(X, Y, by = "t", all = FALSE) )
# [1] TRUE

# all rows from X - left outer join
merge(X, Y, by = "t", all.x = TRUE)
#    t  a  b
# 1: 1  1 NA
# 2: 2  4 NA
# 3: 3  9  9
# 4: 4 16 16

# all rows from both X and Y - full outer join
merge(X, Y, by = "t", all = TRUE)
#    t  a  b
# 1: 1  1 NA
# 2: 2  4 NA
# 3: 3  9  9
# 4: 4 16 16
# 5: 5 NA 25
# 6: 6 NA 36

Update: data.table v1.9.6 introduced the on= syntax, which allows ad hoc joins on fields other than the primary key. jangorecki's answer to the question How to join (merge) data frames (inner, outer, left, right)? provides some examples of additional join types that data.table can handle.

5
  • 9
    Have you read FAQ 1.12? You can always call Y[X] if you want the left outer join of X[Y] and rbind(Y[X],X[Y]) if you want a full outer join
    – mnel
    Oct 8, 2012 at 0:03
  • See my answer for a more data.table approach to the full outer join
    – mnel
    Oct 8, 2012 at 0:18
  • @mnel, I assume your unique() approach below for the full join is preferable to rbind(Y[X],X[Y]), since the rbind would involve copying the table. Is that right? Oct 8, 2012 at 22:03
  • to the best of my knowledge, yes. I haven't tested whether three smaller unique calls are quicker than one large (eg unique(c(unique(X[,t]), unique(Y[,t])) -- this should be more memory efficient as it is only combining two lists that are going to be less than or equal to the number of rows in X and Y.
    – mnel
    Oct 8, 2012 at 22:10
  • 4
    Your question such a good description; I found answers to my questions in your question. Thanks
    – irriss
    May 15, 2013 at 0:05

3 Answers 3

86

To quote from the data.table FAQ 1.11 What is the difference between X[Y] and merge(X, Y)?

X[Y] is a join, looking up X's rows using Y (or Y's key if it has one) as an index.

Y[X] is a join, looking up Y's rows using X (or X's key if it has one)

merge(X,Y) does both ways at the same time. The number of rows of X[Y] and Y[X] usually differ, whereas the number of rows returned by merge(X,Y) and merge(Y,X) is the same.

BUT that misses the main point. Most tasks require something to be done on the data after a join or merge. Why merge all the columns of data, only to use a small subset of them afterwards? You may suggest merge(X[,ColsNeeded1],Y[,ColsNeeded2]), but that requires the programmer to work out which columns are needed. X[Y,j] in data.table does all that in one step for you. When you write X[Y,sum(foo*bar)], data.table automatically inspects the j expression to see which columns it uses. It will only subset those columns only; the others are ignored. Memory is only created for the columns the j uses, and Y columns enjoy standard R recycling rules within the context of each group. Let's say foo is in X, and bar is in Y (along with 20 other columns in Y). Isn't X[Y,sum(foo*bar)] quicker to program and quicker to run than a merge of everything wastefully followed by a subset?


If you want a left outer join of X[Y]

le <- Y[X]
mallx <- merge(X, Y, all.x = T)
# the column order is different so change to be the same as `merge`
setcolorder(le, names(mallx))
identical(le, mallx)
# [1] TRUE

If you want a full outer join

# the unique values for the keys over both data sets
unique_keys <- unique(c(X[,t], Y[,t]))
Y[X[J(unique_keys)]]
##   t  b  a
## 1: 1 NA  1
## 2: 2 NA  4
## 3: 3  9  9
## 4: 4 16 16
## 5: 5 25 NA
## 6: 6 36 NA

# The following will give the same with the column order X,Y
X[Y[J(unique_keys)]]
8
  • 7
    Thanks @mnel. FAQ 1.12 doesn't mention full or left outer join. Your full outer join suggestion with unique() is a great help. That ought to be in the FAQ. I know Matthew Dowle "designed it for his own use, and he wanted it that way." (FAQ 1.9), but I thought X[Y,all=T] could be an elegant way to specify a full outer join within the data.table X[Y] syntax. Or X[Y,all.x=T] for the left join. I wondered why it wasn't designed that way. Just a thought. Oct 8, 2012 at 8:53
  • 1
    @DouglasClark Have added answer, and filed 2302:Add mnel's merge join syntax to FAQ (with timings). Great suggestions!
    – Matt Dowle
    Oct 8, 2012 at 9:44
  • @mnel Is there any way we can impute NA's with 0 when performing X[Y[J(unique_keys)]] ?
    – Ankit
    Feb 26, 2014 at 19:38
  • 2
    @mnel unique_keys <- union(X[,t], Y[,t]) would do the job with lesser keystrokes
    – Selva
    May 12, 2016 at 2:08
  • 13
    what impresses me about data.table documentation is that it can be so verbose, yet remain so cryptic...
    – NiuBiBang
    May 26, 2016 at 16:45
29

@mnel's answer is spot on, so do accept that answer. This is just follow up, too long for comments.

As mnel says, left/right outer join is obtained by swapping Y and X: Y[X] -vs- X[Y]. So 3 of the 4 join types are supported in that syntax, not 2, iiuc.

Adding the 4th seems a good idea. Let's say we add full=TRUE or both=TRUE or merge=TRUE (not sure the best argument name?) then it hadn't occurred to me before that X[Y,j,merge=TRUE] would be useful for the reasons after the BUT in FAQ 1.12. New feature request now added and linked back here, thanks :

FR#2301 : Add merge=TRUE argument for both X[Y] and Y[X] join like merge() does.

Recent versions have sped up merge.data.table (by taking a shallow copy internally to set the keys more efficiently, for example). So we are trying to bring merge() and X[Y] closer, and provide all options to user for full flexibility. There are pros and cons of both. Another outstanding feature request is :

FR#2033 : Add by.x and by.y to merge.data.table

If there are any others, please keep them coming.

By this part in the question :

why not use the merge syntax for joins rather than the match function's nomatch parameter?

If you prefer merge() syntax and its 3 arguments all,all.x and all.y then just use that instead of X[Y]. Think it should cover all the cases. Or did you mean why is the argument a single nomatch in [.data.table? If so, it's just the way that seemed natural given FAQ 2.14 : "Can you explain further why data.table is inspired by A[B] syntax in base?". But also, nomatch only takes two values currently 0 and NA. That could be extended so that a negative value meant something, or 12 would mean use the 12th row's values to fill in NAs, for example, or nomatch in future could be a vector or even itself a data.table.

Hm. How would by-without-by interact with merge=TRUE? Perhaps we should take this over to datatable-help.

9
  • Thanks @Matthew. @mnel's answer is excellent, but my question was not how to do a full or left join, but "Is there a reason only two types of joins are supported?" So now it's a bit more philosophical ;-) Actually I don't prefer merge syntax, but there appears to an R tradition for building on existing stuff folks are familiar with. I had scribbled join="all", join="all.x", join="all.y" and join="x.and.y" in the margin of my notes. Not sure if that is any better. Oct 8, 2012 at 21:41
  • @DouglasClark Maybe join like that, good idea. I posted to datatable-help so let's see. Maybe give data.table some time to settle in, too. Have you got to by-without-by yet for example, and join inherited scope?
    – Matt Dowle
    Oct 8, 2012 at 22:49
  • As indicated in my comment above, I suggest adding a join keyword to , when i is a datatable: X[Y,j,join=string]. The possible string values for join are suggested to be: 1) "all.y" and "right" - Oct 31, 2012 at 12:46
  • 1
    Hi Matt, the data.table library is fantastic; thank you for that; though I think the join behavior (being a right outer join by default) should be explained prominently in the main documentation; it took me 3 days to figure this out. May 12, 2014 at 9:03
  • 1
    @tucson Just to link here, now filed as issue #709.
    – Matt Dowle
    Jun 25, 2014 at 22:41
20

This "answer" is a proposal for discussion: As indicated in my comment, I suggest adding a join parameter to [.data.table() to enable additional types of joins, ie: X[Y,j,join=string]. In addition to the 4 types of ordinary joins, I also suggest to support 3 types of exclusive joins, and the cross join.

The join string values (and aliases) for the various join types are proposed to be:

  1. "all.y" and "right" -- right join, the present data.table default (nomatch=NA) - all Y rows with NAs where there is no X match;
  2. "both" and "inner" -- inner join (nomatch=0) - only rows where X and Y match;

  3. "all.x" and "left" -- left join - all rows from X, NAs where no Y match:

  4. "outer" and "full" -- full outer join - all rows from X and Y, NAs where no match

  5. "only.x" and "not.y" -- non-join or anti-join returning X rows where there is no Y match

  6. "only.y" and "not.x" -- non-join or anti-join returning Y rows where there is no X match
  7. "not.both" -- exclusive join returning X and Y rows where there is no match to the other table, ie an exclusive-or (XOR)
  8. "cross" -- cross join or Cartesian product with each row of X matched to each row of Y

The default value is join="all.y" which corresponds to the present default.

The "all", "all.x" and "all.y" string values correspond to merge() parameters. The "right", "left", "inner" and "outer" strings may be more amenable to SQL users.

The "both" and "not.both" strings are my best suggestion at the moment -- but someone may have better string suggestions for the inner join and exclusive join. (I'm not sure if "exclusive" is the right terminology, correct me if there is a proper term for an "XOR" join.)

Use of join="not.y" is an alternative for X[-Y,j] or X[!Y,j] non-join syntax and maybe more clear (to me), although I'm not sure if they are the same (new feature in data.table version 1.8.3).

The cross join can be handy sometimes, but it may not fit in the data.table paradigm.

6

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.