1

I have a flume sink to write data to aws s3 bucket. Flume config is as below

   aggregator.sinks.s3LogSink.type=hdfs
   aggregator.sinks.s3LogSink.channel=flumeLogAgentFileChannel
   aggregator.sinks.s3LogSink.hdfs.path=s3n://<aws bucket name>
   aggregator.sinks.s3LogSink.hdfs.fileType=DataStream
   aggregator.sinks.s3LogSink.hdfs.writeFormat=Text
   aggregator.sinks.s3LogSink.hdfs.rollCount = 0
   aggregator.sinks.s3LogSink.hdfs.rollSize = 67108864
   aggregator.sinks.s3LogSink.hdfs.batchSize = 1000
   aggregator.sinks.s3LogSink.hdfs.rollInterval = 0

To support the flume s3 sink i also included the below jars

 commons-codec-1.4.jar
 commons-httpclient-3.1.jar
 hadoop-common-2.7.1.jar
 commons-configuration-1.6.jar
 hadoop-auth-2.7.1.jar
 aws-java-sdk-1.10.64.jar
 aws-java-sdk-s3-1.10.52.jar
 hadoop-aws.jar
 emr-core-2.4.0.jar

When i start my flume process .. i get the below error

2016-04-18 18:03:06,326 (hdfs-s3LogSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-04-18 18:03:06,359 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:224)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2623)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/event/ProgressListener
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
        at java.lang.Class.getConstructor0(Class.java:2895)
        at java.lang.Class.newInstance(Class.java:354)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
        ... 18 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 23 more

Can some one please help?

1 Answer 1

0

The correct path to the bucket is of the form s3n://<access-key>:<secret-access-key>@<bucket-name>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.