17

I am seeing errors when starting spark-shell, using spark-1.6.0-bin-hadoop2.6. This is new behavior that just arose.

The upshot of the failures displayed in the log messages below, is that sqlContext is not available (but sc is).

Is there some kind of Derby lock that could be released? Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.

<console>:16: error: not found: value sqlContext
         import sqlContext.implicits._
                ^
<console>:16: error: not found: value sqlContext
         import sqlContext.sql

16/05/25 11:00:00 ERROR Schema: Failed initialising database.
Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.
org.datanucleus.exceptions.NucleusDataStoreException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.


16/05/25 11:06:02 WARN Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient


16/05/25 11:06:02 ERROR Schema: Failed initialising database.
Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@372e972d, see the next exception for details.
org.datanucleus.exceptions.NucleusDataStoreException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@372e972d, see the next exception for details.

Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        ... 134 more
Caused by: java.sql.SQLException: Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        ... 131 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.
6
  • Yes, there is a Derby lock file that prevents two unrelated apps from trying to update the same embedded database simultaneously. Jun 1, 2016 at 20:16
  • Would it make sense to modify or delete that file in order to resolve this behavior? Jun 3, 2016 at 9:52
  • No, that would lead to a corrupted database. The right approach is to figure out which are the two applications that are independently trying to open the same embedded database simultaneously, and why they are both trying to run at once. Either give them each their own DB, don't run them concurrently, or re-configure your system so they can share the DB by using the Derby server rather than the Derby embedded DB configuration. Jun 3, 2016 at 13:43
  • Thanks Bryan. This appears to happen when spark-shell does not exit gracefully (as in a hung session), and then a new session invokes spark-shell. Is there anything one can do in this case to "reset" the embedded DB, i.e. close the hung connection? Jun 7, 2016 at 16:09
  • When Derby is running in embedded mode, the Derby code is running directly in the containing JVM. There is no way to communicate with the Derby code other than from Java code in that JVM, and no way to stop the Derby engine without stopping the the containing JVM. The "connection" to the DB is wholly within that containing JVM. There are other Derby configurations (e.g., client-server) which behave differently, but for the embedded configuration this is the reality of the situation. Jun 7, 2016 at 21:46

6 Answers 6

36

I had a similar problem in a Spark 2.0.0 shell when trying to create a DataFrame, just remove metastore_db/dbex.lck and the problem is fixed.

5
  • better not do this for data safe
    – Pengfei.X
    Dec 10, 2016 at 5:03
  • @Pengfei.X why? and what do you suggest?
    – bachr
    Dec 12, 2016 at 12:12
  • It worked for me too, Not sure why suddenly this error was appeared.. Thanks. Aug 12, 2017 at 7:08
  • 1
    I did delete the metastore_db directory, the problem was solved. This directory is temporary. If you delete it, it will recreate.
    – Ivan Lee
    Sep 28, 2017 at 3:00
  • 1
    Deleting dbex.lck didn't work for me (Spark 2.3.0). I also tried deleting all of metastore_db, but that didn't work either. Running spark-shell with sudo works as andy's said below, but this doesn't seem to be an ideal solution.
    – Logan
    Apr 5, 2018 at 17:31
2

similar to Andy's answer. I had the same issue in windows, and here is the solution:

  1. Run cmd on windows as administrator
  2. navigate to spark home directory
  3. open spark-shell

    c:\park\bin> spark-shell

2

This issue happens because of Metastore_db,it is created when the spark shell is looking for db and start registering it.You can remove metastore_db completely because each and every time it will be created.If you are not able to delete then first delete metastore_db/dbex.lck then you can able to delete the metastore_db folder.I have used spark2.1.10 and faced same issue earlier. enter image description here

1

The best way to resolve the problem is to first restart your system, then go to spark home directory and from there try to run with spark-shell with sudo user.

sudo bin/spark-shell

or if you want to use pyspark instance, the type

sudo bin/pyspark

the problem mainly arises, due to insufficient privileges for mertstore_db.

1

In my case Hive was also started along with Spark. So I closed the Hive server and restarted the Spark shell to make it work.

-3

Issue appears to have been ephemeral as we are no longer experiencing this behavior.

1
  • 1
    I'm having it on both 2.1.0 and 2.1.0
    – Frank B.
    Feb 23, 2017 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.