1

Trying to read files from s3 using hadoop-aws, The command used to run code is mentioned below. please help me resolve this and understand what I am doing wrong.

# run using command
# time spark-submit --packages org.apache.hadoop:hadoop-aws:3.2.1 connect_s3_using_keys.py

from pyspark import SparkContext, SparkConf
import ConfigParser
import pyspark

# create Spark context with Spark configuration
conf = SparkConf().setAppName("Deepak_1ST_job")
sc = SparkContext(conf=conf)
sc.setLogLevel("ERROR")

hadoop_conf = sc._jsc.hadoopConfiguration()

config = ConfigParser.ConfigParser()
config.read("/home/deepak/Desktop/secure/awsCred.cnf")
accessKeyId = config.get("aws_keys", "access_key")
secretAccessKey = config.get("aws_keys", "secret_key")

hadoop_conf.set(
    "fs.s3n.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
hadoop_conf.set("fs3a.access.key", accessKeyId)
hadoop_conf.set("s3a.secret.key", secretAccessKey)

sqlContext = pyspark.SQLContext(sc)

df = sqlContext.read.json("s3a://bucket_name/logs/20191117log.json")
df.show()

EDIT 1:

As I am new to pyspark I am unaware of these dependencies, also the error is not easily understandable.

getting error as

File "/home/deepak/spark/spark-3.0.0-preview-bin-hadoop3.2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 98, in deco
  File "/home/deepak/spark/spark-3.0.0-preview-bin-hadoop3.2/python/lib/py4j-0.10.8.1-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o28.json.
: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
        at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:816)
        at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:792)
        at org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys(S3AUtils.java:747)
        at org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.
2
  • What is the error that you are seeing ? What have you tried till now to resolve it ? Do add all those details for someone to help you better. Dec 1, 2019 at 6:47
  • @JayadeepJayaraman I have added the error please check. I am new to pyspark Dec 1, 2019 at 7:17

2 Answers 2

13

I had the same issue with spark 3.0.0 / hadoop 3.2.

What worked for me was to replace the hadoop-aws-3.2.1.jar in spark-3.0.0-bin-hadoop3.2/jars with hadoop-aws-3.2.0.jar found here: https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0

2
  • Im using a linux machine, i dont already have the hadoop-aws-3.2.1.jar or any version of hadoop-aws jar. How do i get this jar, and set it up?
    – user14281567
    May 3, 2021 at 15:54
  • Is just just wget repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.0/… and move the file to jars folder or is there something more that i should do?
    – user14281567
    May 3, 2021 at 15:56
1

Check your spark guava jar version. If you download spark from Amazon like me from the link (https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-3.0/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3.tgz) in their documentation. You can see the include guava version is guava-14.0.1.jar and their container is using guava-21.0.jar

I have reported the issue to them and they will repack their spark to include the correct version. If you interested in the bug self, here is the link. https://hadoop.apache.org/docs/r3.2.0/hadoop-aws/tools/hadoop-aws/troubleshooting_s3a.html#ClassNotFoundException:_org.apache.hadoop.fs.s3a.S3AFileSystem

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.