1

I am trying to execute following query on Spark in yarn-client mode.

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi     --master yarn     --deploy-mode client         $SPARK_HOME/examples/target/scala-2.10/spark-examples*.jar     10

When I execute above query my application gets stuck on following

16/07/13 17:14:28 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

16/07/13 17:14:28 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1468430067384 final status: UNDEFINED tracking URL: http://hadoop-master:8088/proxy/application_1468428769910_0002/ user: nachiket

16/07/13 17:14:29 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

16/07/13 17:14:30 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

16/07/13 17:14:31 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

16/07/13 17:14:32 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

I have already implemented most of suggestions mentioned on following link :

Application report for application_ (state: ACCEPTED) never ends for Spark Submit (with Spark 1.2.0 on YARN)

I am still facing the same issues . Are there any other solutions than above mentioned link ?

Finally Job fails with following clause

    client token: N/A
         diagnostics: Application application_1468455134412_0001 failed 2 times due to Error launching appattempt_1468455134412_0001_000002. Got exception: org.apache.hadoop.net.ConnectTimeoutException: Call From sclab103/104.239.213.7 to 104.239.213.7:60640 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=104.239.213.7/104.239.213.7:60640]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
        at org.apache.hadoop.ipc.Client.call(Client.java:1479)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy82.startContainers(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy83.startContainers(Unknown Source)
        at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
        at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:250)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=104.239.213.7/104.239.213.7:60640]
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
        at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
        at org.apache.hadoop.ipc.Client.call(Client.java:1451)
        ... 16 more
. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1468455280498
         final status: FAILED
         tracking URL: http://hadoop-master:8088/cluster/app/application_1468455134412_0001
         user: sclab
3
  • you have to check memory configuration of your yarn cluster, how much is being allocated to your resource and node managers .
    – nat
    Jul 14, 2016 at 17:47
  • Hi @nath , I am running only Spark Pi job and I am also making sure that no other job is running on the cluster. following is my yarn-site memory settings : yarn.nodemanager.resource.memory-mb 3072 yarn.scheduler.minimum-allocation-mb 512 yarn.scheduler.maximum-allocation-mb 3072 One important thing that i have noticed is that i am not able to make connection with yarn through spark I tried spark-shell --master yarn-client but it gives same error as mentioned above.
    – npaluskar
    Jul 14, 2016 at 18:44
  • please check if your yarn cluster is up, check your yarn resource and node logs , it seems to be connectivity issue.
    – nat
    Jul 14, 2016 at 21:21

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.