I have two scenarios shown below:
scala> val dfA = sqlContext.read.parquet("/home/mohit/ruleA")
dfA: org.apache.spark.sql.DataFrame = [aid: int, aVal: string]
scala> val dfB = sqlContext.read.parquet("/home/mohit/ruleB")
dfB: org.apache.spark.sql.DataFrame = [bid: int, bVal: string]
scala> dfA.registerTempTable("A")
scala> dfB.registerTempTable("B")
1 .Left Join with Filter in WHERE
sqlContext.sql("select A.aid, B.bid from A left join B on A.aid=B.bid where B.bid<2").explain
== Physical Plan ==
Project [aid#15,bid#17]
+- Filter (bid#17 < 2)
+- BroadcastHashOuterJoin [aid#15], [bid#17], LeftOuter, None
:- Scan ParquetRelation[aid#15,aVal#16] InputPaths: file:/home/mohit/ruleA
+- Scan ParquetRelation[bid#17,bVal#18] InputPaths: file:/home/mohit/ruleB
2. Left Join with Filter in ON
sqlContext.sql("select A.aid, B.bid from A left join B on A.aid=B.bid and B.bid<2").explain
== Physical Plan ==
Project [aid#15,bid#17]
+- BroadcastHashOuterJoin [aid#15], [bid#17], LeftOuter, None
:- Scan ParquetRelation[aid#15] InputPaths: file:/home/mohit/ruleA
+- Filter (bid#17 < 2)
+- Scan ParquetRelation[bid#17] InputPaths: file:/home/mohit/ruleB, PushedFilters: [LessThan(bid,2)]
Question
In either cases, Catalyst
had information that from table B, only B.bid
(bid#17) is required. Why did entire table scan was required in WHERE
case. The projection
columns for table B are implicit and deterministic.
Note: This is a watered-down example from a production issue. Spark version - 1.6.2.