BRAIN DUMP: Conventional Routing in RabbitMQ | Dru Sellers

:

Thinking at Progress Cafe

As I sit here and ponder how we are going to implement RabbitMQ in MassTransit I thought it would be nice to share my thoughts. Below is my brain dump on how we are thinking we will approach RabbitMQ.

Please give us feedback!

TL;DR; – skip to the picture below / take 4.

-d


Thinking about how to do conventional routing, with RabbitMQ and MassTransit.

Quality Goals:

  • Contain complexity, the solution should not bust at the seems as the system grows. ie no one piece should just explode and become unmanageable.
  • allow dev / prod networks

The first thing that I did, is I went and looked at how some of the other projects were approaching it. Below is my attempt to summarize them.

Current MassTransit Thinking

  • Publishes to Exchange [Fanout] <TypeName>
  • No Routing Key needed
  • listens on queues named for the service

EasyNetQ

  • Publishes to Exchange [Direct] <TypeName>
  • Routes Key <TypeName>

Symbiote

  • is not ‘as’ conventional, alex please correct me
  • exchanges are made on a per bus response id (aka agent?)
  • a queue is made per agent / message type
  • Symbiote has a ‘bus’ api and a ‘mesh’ api

I then did a bunch of searching for RabbitMQ routing articles, lurked on #rabbitmq, and in general harassed @phatboyg.

Articles for further enlightenment found during exploration

Routing Topologies for Performance and Scalability

Exchange-to-Exchange bindings in RabbitMQ

Very fast and scalable topic routing – part 1part 2

Broker vs Brokerless

RabbitMQ convention take 2

  • Publish to a single Exchange (based on environment name like ‘develop’)
  • RoutingKey: A combination of the type and its interfaces*
  • You would bind to the exchange with routing key #.MessageName.#

This approach seem awfully naive and completely misses out on the power of rabbit routing topologies. As well as overloading the single exchange with a butt load of routings.

So one exchange isn’t enough. So what if we were to additional use the assembly name for some routing point?

RabbitMQ convention take 3

  • publishes to an exchange for each assembly of messages
  • connects to a vhost for network segmentation like develop
  • RoutingKey: still a combination of the type and its interfaces
  • would bind to the exchange with #.MessageName.# like Take 2

take 3 just seems silly, and was quickly dismissed. still missing out on the routing topologies. Its at this point that exchange-to-exchange bindings really sink in.

RabbitMQ convention take 4

  • networks are segregated by vhosts
  • we generate an exchange for each queue so that we can do direct sends to the queue. it is bound as a fanout exchange
  • for each message published we generate series of exchanges that go from concrete class to each of its subclass / interfaces these are linked together from most specific to least specific. This way if you subscribe to the base interface you get all the messages. or you can be more selective. all exchanges in this situation are bound as fanouts.
  • the subscriber declares his own queue and his queue exchange – he then also declares/binds his exchange to each of the message type exchanges desired
  • the publisher discovers all of the exchanges needed for a given message, binds them all up and then pushes the message into the most specific queue letting RabbitMQ do the fanout for him. (One publish, multiple receivers!)
  • control queues are exclusive and auto-delete – they go away when you go away and are not shared.
  • we also lose the routing keys. WIN!

A Visual:

The consumer can subscribe at whatever level makes the most sense, yet the routing will stay constant otherwise. By having an exchange bound to the queue we also minimize binding churn as the agent comes up and down.