A short while back I wrote a piece, Why Message Queues Suck. The gist of the piece is that for the same labor and overhead required to implement and maintain an inter-service notification architecture using message queues, you can just as easily implement inter-service notification by Direct To Endpoint communication. (Please see Figure 1 below.)
As you might imagine, the piece caused quite a stir, particularly on Reddit. Many readers did not buy the idea for a variety of reasons. One area of disagreement was around fail safety and high availability. A reader going by the handle, singlecoilpickup commented:
I feel like the topic is missing some discussion of high availability and fail over [sic]. Breaking the argument down, yes, the world is simpler if you just send a message to an http endpoint in your service. What makes that a bit less realistic though is guaranteeing that service is reachable and working as intended.
Message queues are relatively easy to make highly available, even if you're not using something like SQS. Being able to send something to a highly available queue provides a better guarantee that the message will get processed, especially in the event of a failure.
Another area of disagreement was with support for retries, as articulated by the reader, threemux:
As others have pointed out, there is no such thing as a “simple POST”. This article contains no mention of retry logic in the event of failed consumers. Or worse, slow consumers.
And the reader, washtubs:
I'm surprised there was basically no mention of guaranteed delivery which is one of the sticking points of electing to use a message queue.
Finally, the reader vkgfx refuted the assertion of cost savings:
…all those costs I just described are data transfer out of AWS. It's not always true, but if he's building his whole system on an AWS stack, any competent dev will try to have EC2 instances in the same region as their connecting SQS queues. So how much do those 1 billion messages cost in transfer in this case? $0.
Each of the points raised was valid and worthy of response, to which I made a few. I needed to be careful. It's easy to get into to a flame war over fine technical points. It goes with the territory. Most developers are very smart and some are very very smart. One wrong step and you can end up on the receiving end of a merciless and sometimes painful retort.
But I was surprised. This discussion was different. It was an exchange of ideas, not a battle of unsubstantiated opinion. Reason and civility prevailed. And because the discussion was reasonable, I was willing to participate.
In his book, The Fifth Discipline, Peter Senge, a thought leader about learning organizations, talks about creative tension. Creative tension is the intersection of vision and reality. Applying an idea to the reality of our situation will cause a tension from which a new, better idea will emerge. Creative tension is at the core of innovation. Was this Reddit page full of creative tension? Yes. Were better ideas emerging? Yes, seems that way. Could the same be said were the discourse to be full of insults and cynical barbs? I don't think so.
So, where did we end up?
The reader Xiopos summed it up concisely, “As usual, there's a trade-off.”
As I mentioned in the article, I was and am to a certain degree a big supporter of message queues, if for no other reason than they promote event driven architectures. Understanding how to work with events is essential when doing asynchronous programming in the modern day Internet/Cloud. Using message queues is a fast track to this understanding. Yet, message queues are not the only means by which to work with events. Arguably Direct to Endpoint is an alternative. Still, there will always be tradeoffs. Each situation is different in terms of requirement, time and budget. To my thinking there is no black and white, only shades of gray. The best solution will come about by the iteration of ideas based on acquiring new information.
What do I think is the most important takeaway from my experience with having the ideas in my article jostled about on Reddit? It's this: having observable proof that civility works when discussing conflicting ideas.
Let's face it, technologies come and go. The technical landscape we will occupy in 5 years will be completely different from the one we have now. That's the way the business is. Change is fast and furious. Conflicts will arise. But, we have a choice. When faced with the presentation of an idea that conflicts with our thinking we can fight it out, leaving the losers thrown to the side of some sort intellectual highway or we can discuss our conflicting ideas in a reasoned, civil manner that promotes a willingness participate in the creative tension at hand. Remember, it's creative tension to produces innovation. Intellectual fistfights just promote bloody noses. Me? I'll take innovation every time. And yes, given what I learned from the discussion on Reddit, message queues might not suck.
