4

Do you know how to integrate Kubernetes with existing Yarn cluster?

Below page explains the features that Kubernetes Yarn integration offers.

https://hortonworks.com/blog/docker-kubernetes-apache-hadoop-yarn/

However, the tutorial it refers only gives instructions to install a local cluster.

Alternative or a second question would be:

Is that advisable / rational to run Kubernetes cluster in the same physical cluster that Hadoop runs?

Thanks in advance...

1 Answer 1

0

The hortonworks solution seems odd to me. Kubernetes manages docker containers on physical machines, Yarn schedule jobs a cluster (built by a set of linux boxes). The nature way of integration should be letting yarn running on docker containers managed by Kubernetes. Why the opposite?

IMO, the right solution is not integrate kubernete to your existing yarn cluster. Instead, you should rebuild your cluster with micro-service oriented architecture, which means every service (yarn is one of them) is running in docker containers managed by kubernetes. Long term, this will be a more manageable, maintenance-save approach for a big data cluster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.