2

I'm trying to use H2O.ai's sparkling-water via spark packages. I'm following their guide: https://github.com/h2oai/sparkling-water#use-sparkling-water-via-spark-packages I'm on Hortonworks HDP 2.4 with Scala 2.10 and Spark 1.6.1. I put the following in my terminal:

spark-shell --packages ai.h2o:sparkling-water-core_2.10:1.6.8,ai.h2o:sparkling-water-examples_2.10:1.6.8

However, it complains two artifacts not found. Here're the details:

:: problems summary ::
:::: WARNINGS
        [NOT FOUND  ] com.google.guava#guava;16.0.1!guava.jar(bundle) (1ms)
==== local-m2-cache: tried

  file:/root/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar

    [NOT FOUND  ] com.google.code.findbugs#jsr305;3.0.0!jsr305.jar (1ms)

==== local-m2-cache: tried

  file:/root/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.jar

    ::::::::::::::::::::::::::::::::::::::::::::::

    ::              FAILED DOWNLOADS            ::

    :: ^ see resolution messages for details  ^ ::

    ::::::::::::::::::::::::::::::::::::::::::::::

    :: com.google.guava#guava;16.0.1!guava.jar(bundle)

    :: com.google.code.findbugs#jsr305;3.0.0!jsr305.jar

    ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: com.google.guava#guava;16.0.1!guava.jar(bundle), download failed: com.google.code.findbugs#jsr305;3.0.0!jsr305.jar]
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1068)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:287)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Anyone has any experience how I can fix it? Thanks a lot.

3 Answers 3

1

Removing ~/.m2 and ~/ivy2 fixed the issue with using ai.h2o:sparkling-water-core_2.10:1.6.8

1
  • I guess you mean ~/.ivy2 as it is a hidden directory.
    – kar09
    May 30, 2021 at 10:33
0

If you look for the error "download failed: com.google.guava#guava;16.0.1!guava.jar" around, you will see it is mostly an environment specific issues where packages could not be downloaded during the application launched.

Your solution could be to download/install both the failed downloaded package separately something as below to get you going:

$ mvn install com.google.guava:guava:21.0:get -DrepoUrl=https://mvnrepository.com
0

It looks like lowering the version of the jar files

spark-shell --packages ai.h2o:sparkling-water-core_2.10:1.6.1,ai.h2o:sparkling-water-examples_2.10:1.6.1

or using the fatjar suggested in Section 5.1.1 of Sparkling Water Booklet

spark-shell --jars [your path]/sparkling-water-1.6.8/assembly/build/libs/sparkling-water-assembly-1.6.8-all.jar

solves the issue. Where sparkling-water-1.6.8 is downloaded from H2O

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.