2

Originally I had a single rabbitmq node working just fine with a request/response client interaction.

I'm now changing to a cluster and trying to run the very same request/response operation. It blows up spectacularly.

I've setup 2 hosts as part of the rabbitmq cluster. I was having so much trouble that I reverted to using the masstransit sample code.

This also seems to have trouble. I end up getting a recurring exception on my request service:

--- End of stack trace from previous location where exception was thrown --
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(...)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(...)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwai
ter.GetResult()
   at MassTransit.Pipeline.Filters.RescueReceiveContextFilter`1.<MassTransit-Pip
eline-IFilter<MassTransit-ReceiveContext>-Send>d__5.MoveNext()
Returning name for 45
Rescuing exception
MassTransit.EndpointNotFoundException: The endpoint address specified an unknown
 host: rabbitmq://erinome:5672/bus-ERINOME-Client.vshost-1pooyygnzw6bmox5bdjwjoy
mnw?durable=false&autodelete=true&prefetch=4
   at MassTransit.RabbitMqTransport.RabbitMqSendTransportProvider.GetSendTranspo
rt(Uri address)
   at MassTransit.RabbitMqTransport.RabbitMqSendEndpointProvider.<GetSendEndpoin
t>d__5.MoveNext()

I'm running both client and request service from the same host in the simplest configuration I can to observe the issue.

Some sample configuration for the client:

key="RabbitMQHost" value="rabbitmq://erinome"
key="ServiceAddress" value="rabbitmq://erinome/request_service"

and the config for the request service:

key="RabbitMQHost" value="rabbitmq://localhost"
key="ServiceQueueName" value="request_service"

My issue is probably lack of knowledge with clusters and the endpoint not found exception points to some message routing issue. Any help appreciated.

3
  • So the service is using a local RabbitMQ server (localhost)? Seems like you would need to make sure the erinome host name resolves to that same local host in that case. Mar 6, 2016 at 16:39
  • If I change everything to use localhost then it is all sweet, client and server communicate nicely, even if they are connected to different hosts of the cluster.
    – bchap
    Mar 8, 2016 at 1:12
  • If I change to use the host name instead of localhost, then I get communication issues. All hostnames can be resolved in hosts file. The context looks to have a response address containing "localhost" even though it should be sent to a remote host. Log of response on host herse to a request from host erinome DEBUG [17] (null) - SEND:rabbitmq://**localhost**:5672/bus-ERINOME-program-1pooyygnzw6bmco7bdjwp4rcf7?durable=false&autodelete=true
    – bchap
    Mar 8, 2016 at 1:32

1 Answer 1

4

If you are using a cluster, you are typically trying to ensure high availability of message queues in the presence of a node failure. In this case, the nodes should be standalone machines that are not actually running services - since those services would likely go down with the node itself.

Clustering also is done so that queues can be replicated across the available nodes - a form of high-availability - which ensures that services are able to recover the messages from a down node when that node fails.

There are extensive articles on clustering RabbitMQ, including the official RabbitMQ site and several other sites that have tested the cluster stability of RabbitMQ using various failure mechanisms (Jespen comes to mind here).

When using a cluster and distributed nodes, it's important to use a cluster DNS name that resolves to the active or primary node in the cluster -- ensuring that messages are properly delivered to the cluster and consumers on the network.

The best case is to have a couple of nodes that are in a cluster, with a failover load balancer keeping all consumers connected to node 1 until it crashes, and then fail traffic over to node 2. If node 1 returns to service, it can be manually rejoined to the cluster and traffic moved back over to node 2. You can have three nodes as well, but increased node counts increases replication time so keep that in mind.

In most of these situations, localhost is not your friend and not recommend. Also, if you aren't replicating queues, you could end up with messages existing on only a single node which would require consumer services to exist which connect to every node where a message may be sent. Very confusing and hard to manage in production.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.