-1

I followed the http://stormcrawler.net/getting-started/ guide to generate the jar file for topology.When i run the topology by using storm command in readme file , i get following error in execution of FetcherBolt. I have storm Storm 1.1.0.2.6.4.0-91 installed in a Hortonworks cluster. I get same exception regardless of -local or distributed mode. I got following exception

java.lang.NoSuchMethodError: org.apache.commons.logging.impl.LogFactoryImpl.handleThrowable(Ljava/lang/Throwable;)V at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:568) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) at org.apache.http.conn.ssl.AbstractVerifier.(AbstractVerifier.java:61) at org.apache.http.conn.ssl.AllowAllHostnameVerifier.(AllowAllHostnameVerifier.java:44) at org.apache.http.conn.ssl.AllowAllHostnameVerifier.(AllowAllHostnameVerifier.java:46) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.(SSLConnectionSocketFactory.java:146) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.getDefaultRegistry(PoolingHttpClientConnectionManager.java:115) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.(PoolingHttpClientConnectionManager.java:122) at com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol.(HttpProtocol.java:76) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.digitalpebble.stormcrawler.protocol.ProtocolFactory.(ProtocolFactory.java:60) at com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:738) at org.apache.storm.daemon.executor$fn__9635$fn__9648.invoke(executor.clj:794) at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748)

1 Answer 1

0

This is probably due to a conflict between the version of commons-logging inherited from the httpclient library and one which is put on the classpath by the Hortonworks version of Apache Storm.

[INFO] +- org.apache.httpcomponents:httpclient:jar:4.5.3:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.6:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  \- commons-codec:commons-codec:jar:1.9:compile

You could try using a different protocol implementation by setting

  http.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"
  https.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"

in the crawler-conf.yaml file. Note that this does not guarantee that the call to commons-logging won't happen somewhere else. Ideally, you'd want to resolve the dependency problem e.g. by making sure that Hortonworks uses the same version as the one needed by StormCrawler. We support only the Apache distribution of Storm.

1
  • making entry in yaml resolved the issue.our crawler will be deployed on existing infrastructure which we have so will need to see if we can upgrade storm which is there in hortonworks,
    – chans.best
    Apr 9, 2018 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.