1

I have an HPC cluster application where I am looking to replace MPI and our internal cluster management software with a combination of Kubernetes and some middleware, most likely ZMQ or RabbitMQ.

I'm trying to design how best to do peer discovery on this system using Kubernetes' service discovery.

I know Kubernetes can provide a DNS name for a given service, and that's great, but is there a way to also dynamically discover ports?

For example, assuming I replaced the MPI middleware with ZeroMQ, I would need a way for ranks (processes on the cluster) to find each other. I know I could simply have the ranks issue service creation messages to the Kubernetes discovery mechanism and get a hostname like myapp_mypid_rank_42 fairly easily, but how would I handle the port?

If possible, it would be great if I could just do:

zmqSocket.connect("tcp://myapp_mypid_rank_42");

but I don't think that would work since I have no port number information from DNS.

How can I have Kubernetes service discovery also provide a port in as simple a manner as possible to allow ranks in the cluster to discover each other?

Note: The registering process knows its port and can register it with the K8s service discovery daemon. The problem is a quick and easy way to get that port number back for the processes that want it. The question I'm asking is whether or not there is a mechanism as simple as a DNS host name, or will I need to explicitly query both hostname and port number from the k8s daemon rather than simply building a hostname based on some agreed upon rule (like building a string from myapp_mypid_myrank)?

1
  • You put down the service port when you define the service. It's not a variable, the port is whatever number you put down. There are many ways to inject and you get the port number with it but fundamentally it's what you put down on the service definition May 21, 2018 at 21:08

2 Answers 2

2

Turns out the best way to do this is with a DNS SRV record:

https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services https://en.wikipedia.org/wiki/SRV_record

A DNS SRV record provides both a hostname/IP and a port for a given request.

Luckily, Kubernetes service discovery supports SRV records and provides them on the cluster's DNS.

1

I think in the most usual case you should know the port number to access your services.

But if it is useful, Kubernetes add some environment variables to every pod to ease autodiscovery of all services. For example {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT. Docs here

3
  • 1
    The registering process knows the port number when registering it. That's not the problem. The problem is getting the port number back. The environment variable feature of k8s doesn't help, as it is subject to race conditions and has no reconnect capability.
    – stix
    May 21, 2018 at 21:43
  • 1
    Then I think the only way to do it is another container with a daemon that calls the Kubernetes API for changes in services, and updates other pods in consequence. May 22, 2018 at 7:39
  • Bear in mind though @IgnacioMillán that this is exactly what plugins like CoreDNS do and does not depend on ordering of creation of services vs pods.
    – Oliver
    May 6, 2022 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.