Why messaging queues suck


The word around the water cooler is that  a queue has yet to be created that I don’t like. Whether it’s RabbitMQ, AWS SNS/SQS, or Google Cloud Pub-Sub,  regardless of the implementation, I love queues to death, gobble, gobble…I’ll eat ‘em up. I mean, what’s not to like?

Not too long ago at a due diligence review, I was presenting my idea for a mission-critical enterprise architecture. The Pub-Sub pattern played a critical role in my thinking. So I did my dog and pony presentation and things seemed to have gone swimmingly well, then later that day one of the attendees at the presentation stopped by my desk and told me, “I like your thinking, but I gotta tell you, I hate queues. I think they suck.”

I was dumbstruck. My world shook. I felt as if I were a five year old and someone had just told me there was no Santa Claus, and I could not imagine a world with no Santa Claus.

My immediate reaction was to flip the bozo bit and dismiss his comment as one made by a guy who had no idea as to what he was talking about, but, I knew the background of the person. He was no dope and he had a boatload of experience. He’s worked in telecom for a very long time, on very large systems. Given his background and expertise, I’d be dumb not to consider his position. Going against every impulse had to defend my ego, I said, “Oh, why?”

And he told me.

Using a Queue is Lazy

“Basically using a message queue to facilitate interservice communication is lazy,” he said. “You should just have one service send a HTTP POST to the other service that wants the information. For a little more work, you get a lot more bang. Let me show you on the whiteboard.”

Figure 1 shows what he drew.


Figure 1: A typical Pub-Sub pattern using a message queue subscribed to a topic.

“In a typical Pub-Sub pattern you have a service that sends a message to a topic (1). If there are no queues subscribed to the topic, the messages accumulate, eating storage resource. Yes, you can configure a topic to delete a message after a time, but still, the topic is responsible to store the message.

“Luckily, in this case I’ve diagrammed, we have a queue subscribed to the topic. The topic could be a list that’s populating quickly such as real-time stock transactions for a stock brokerage firm. The topic will send a copy of the message to the subscribers it knows about (2). For example, the brokerage’s accounting system as well as another system belonging to the brokerage’s official auditor. Then, once all subscribers get the message, the topic will flush the message.

“Now we have the message sitting in the queue waiting for the Service bound to the queue to pull the message (3). That’s a lot of work. Not only do we have to devote resources to getting the message from the publishing service to the ultimate consuming service(s), but the consuming service has to create the queue and then subscribe the queue to the topic.

“For the same amount of work or less, the creator of the message can send it directly to the interested service, as I show in Figure 2.”


Figure 2: Passing a message directly to a service using the HTTP POST method.

Not one to take things at simple assertion, I decided to put up some opposition to the idea.

Things are Not as They Seem

“Look”, I say. “The virtue of the Pub-Sub pattern is that the publisher of the message knows nothing about the consumer of the message. The publishing service just sends the message to the topic and that’s that. Any subsequent action is unknown to the publisher. Millions of queues might be subscribed to the topic. The queues can drop in and out as they need to. The elegance is that the publishing service does not have to handle that dynamic nature that goes with the registration of subscribers. The burden of subscriber registration is left to the topic.”

“My friend”, he says. “Things are not as they seem. Let’s get real world here.”

He continues drawing on the whiteboard with his back facing me. “Most enterprises that operate at large scale do not roll-their-own in terms of topic/queue hosting. They use a service provider such as AWS. Every topic in play costs money and the storage on those topics cost money. Thus, you are absorbing the storage costs of messages until all subscribers pull the messages down. That’s one little hidden factor, the storage costs you incur. Here is another: the publisher of the messages is incurring a cost for registering consumers that want your information. Yes, you are incurring the expense indirectly, but there is a real expense. You’re not going to let just anybody subscribe to a topic. At the least, if you are using AWS, you are going to have to grant interested consumers rights to subscribe to the topic. There is no magic here. That process needs to be managed, at least in terms of approval. So for the same labor, why not have services interested in your information register to your service directly? Take a look what I just wrote out in Figure 3.


Figure 3: A way to have an interested service register to a service to get information.

“Registering a consumer directly to a publishing service requires telling the publishing service the URL to where published messages are to be sent. And yes, you might need to include some security information in order for the publisher to be able to access the URL,  but this is a common pattern.

“Of course you do have to implement a way to do subscriber list management, but you’re paying for that already. For the half-day it will take a developer to create a way to register list members and send information to those members, you’re making that expense back in the savings you incur by not having to store any messages at all. You’re shifting that message handling and storage cost to the consumer.”

I thought about what he was saying. As much as I was resisting, he was beginning to make sense. So to drive the point in a little deeper, he went to another whiteboard and drew out a table, shown below:

Table 1: A comparison features between Using Pub-Sub and Direct Access to an Endpoint.

Using Pub-SubDirect Access to Endpoint
Consumer needs to know location of get messageYesYes
Consumer needs to register to sourceYesYes
Source knows about consumersYesYes
Allows late bindingYesYes

“Take a look at this table”, he says. “Whether you used Pub-Sub or Direct Access To an Endpoint, parties interested need to know the location of a source. In the case of Pub-Sub, the consumer needs to know the topic address and have rights to the address. In Direct Access To an Endpoint the consumer needs to know where the messages are coming from to register to the publishing service to get information. With Pub-Sub, you need to register, too, to a topic. But let’s face it, no matter what, your consumers are registering to an address. It’s a wash.

You might say that Pub-Sub is agnostic in terms of knowing about who is consuming messages. It may seem that way, but really when you think about it, the publisher does have knowledge. That knowledge resides in the topic. Remember, you own the topic and are responsible for granting access to topic. Of course you know who your consumers are.

As far as the benefit of late binding goes, in Pub-Sub you subscribe to the topic, in Direct Access To an Endpoint you register to the service sending the messages. Don’t want the information anymore? Unsubscribe or unregister. Same labor.”

I could feel my mind slowly changing. Obviously he had thought things through a great deal. But, still I had to ask, “Are you telling me that queues are a waste of time and I should hate them too?”

“Not exactly”, he replied. “There is a place for queues.”

A Place for Queues

“The place for queues is in the consumer,” he says. He went back to the whiteboard and scribbled away, creating Figure 4.


Figure 4: Using a queue within a Service.

He continued, “Remember, the fundamental benefit of not using topics or queues for interservice communication is that we don’t have to incur resource costs. We push all of it onto the consumer. However, a service that consumes our information might want to use a queue, particularly if that service has a lot of parallel processes using the information we are delivering. The important part here is that the consumer of the information is incurring the costs associated with its need. Your job is to deliver information. The service’s job is to consume it.”

“Makes sense”, I replied, feeling humbled.

What It All Means

As I stated at the beginning of this piece, I really like queues and topics. Changing my mind is not easy. I think the same can be said for most of us who have been doing this sort of work for a long time. Yet having the ability to examine a new idea and adjust accordingly is the sign of a mature professional, even if it requires that we say those humbling words, “You are right and I am wrong.”

I approach using queues differently now. I don’t hate them. In fact, I still like them. What’s changed is the way I like them. It’s like when I first learned to use a hammer—everything was indeed a nail. Now, I can tell a nail from a screw or a grommet, and I know enough to use the right tool for the situation.

Sometimes growth is hard. But, it beats the alternative.


This article first appeared on ProgrammableWeb.


We'd love to hear your opinion on this post

14 Responses to “Why messaging queues suck”

  1. It sounds like your friend has consuming systems that never, ever fail.

    “Thus, you are absorbing the storage costs of messages until all subscribers pull the messages down.”

    If they can’t immediately pull it down, the assumption is also that you couldn’t deliver the message to them. The difference is with the topic they will pull it down when their system comes back up.

    Integration, in my mind, is all about being the playground recess monitor. Yes, the publishers job is to deliver information. However, if the subscriber doesn’t get the information, that could be bad for both.

    I’ll stick with topics and queues 🙂

    • That is why they mention this – “The place for queues is in the consumer”

      Surely, the consumer can keep queuing the messages while its system is down and then pull those in when the system comes back up.

      Consumers maintaining their own queues could be a more robust solution. What happens when your messaging queue goes down? All consumers lose access to the messages. If only one customer’s queue goes down, the others would still function normally.

      • I agree with Steve: you add a heavy overhead on the publisher (let’s say I’m buying a product on a retail store web, and an email is sent with the confirmation of the purchase, plus a message to the billing system, another one to the shipping system and so on) and makes that point a SPOF. When this is part of my critical path, I don’t want to risk it by coupling to a lot of external services which I won’t be necessarily able to control (yes, the topic itself it’s another point of failure, but queues that handle loads of messages should be properly HA-ready as it’s something I can handle), and to be as quick as possible to go back to the client. The only valid point I see here is that sometimes working with queues are part of an over-engineering solution.

      • The fact that place for quees is in consumer doesa not mean it solves problem of missed delivery. If consumer is down then consumer queue may aslo be down. Your guarenteed delivery is not on consumer side. In addition you make producer for messages havily depndent on consumer responsivness. That means you have to implement timeout policies and that also means there is no any guarantee on delivery. Not the best approach. I disagree that queue is on consumer side. The is short sighted approach from realistic operations point of view. that queue on consumer side is pureley to solve local consumer problems with paralellism, response time, deferred processing etc. bit it does not have much to do with contract between producer and consumer on any guarentees. I will give you realistic example of financial payment instructions. There must be delivery guarentee. At the same time liquidity of the firm is tracked in dashboards. If you miss any notification on state of processing payment instructions then it is absolutely unacceptable solution. Bear in mind that dashboards may be down for maintenance (eg. prod release) and if they were consumers with internal queues they could be down as well. So that puts in sitiuation that this queue is either separte instance not bound to consumer (not on consumer side, but dedicated to consumer) or it should be simply guarenteed delivery topic on producer side. One has to remember that in the worlld of guarenteed devlivery consumers may be down for period of time while gurantees are on producer’s side.

  2. I feel that despite trying to appear balanced and well informed, this post also has sensationalist tendencies, which make it challenging to take this seriously.

    Here is what’s going on here:

    1) In micro-services architecture there is a fundamental need for services to talk to each other. They can talk synchronously, typically via HTTP/HTTPS, or asynchronously, by definition meaning — a queue, with potential routing and fan-out capabilities.

    2) which approach you choose does not depend on “someone proclaiming that queues suck”. It’s like saying “air sucks”, or “water sucks”. We need queues just like we need water, and maybe the point of this article is to say that *sometimes* we over-do it with queues, when we could have used direct synchronous HTTP comms. If THAT is the point, I agree with it. But I don’t agree with a blanket “sucks”.

    3) when doing synchronous bi-directional communication between two services over HTTP, both must be up to receive notifications from each other. One service being down means both are effectively down, or at the very least loosing data because they can’t synchronously deliver it to the other service. You could build a retry logic, but that itself is commonly implemented as a queue.

    4) what the post is saying — is that when using queue *within a service boundary*, i.e. for internal implementation, then it’s “OK” to do so, but using queues *between* services is somehow not OK. Again, I am not convinced.

    5) By choosing to use a queue the implementation becomes more complex, we are spending more on storage, and RAM. But, hopefully, you are not just spending it because you can, but because of the benefit of the having less coupled systems architecture, more tolerance to failures, ability to split a portion of the traffic to different servers for scaling (fan out), and the ability to develop and subscribe new applications of the existing data streams, that enhance the overall system with novel capabilities, all without touching the direct knowledge of the producer.

    My 2c.

  3. I think few important concepts are lost.
    1) Ability to scale – you don’t care about balancers, etc, you just add more producers/consumers to the topic, that’s it
    2) HTTP transport is slow compared to alternative transport message queue can provide.
    3) Queue clients fit better for asyncronious
    In other words, message queue is a more flexible approach

  4. I’m melting because heat here, so i might didn’t read too carefully.
    How you want to deal with scenarios where your subscribers are not single service, but multiple clients and you even don’t know which one are currently connected? how you want to deal with that?
    Will you just create another service which clients will be polling for messages? Which sounds like hacked version of PubSub. Or you have another solution for that?

  5. I think you need to know how and when to use queues. If 3rd party application send new data to http rest then the rest api is down or out of service and needs to bounce the server then what happen is the request or process of new data is void and need to wait the rest api service up and running. In order to avoid this scenario messaging Queue is the great solution you don’t need to worry your counter part app if it’s down or not running the queue is always there and waiting for pickup. That’s why its called the data broker to avoid waiting each other.

  6. This is bad advice and presumes that “Consumers” are some magical creatures that don’t go down, and real world doesn’t have requirements for reliable messaging, or ordered message delivery. It’s an interesting discussion of perspectives, but the (false) “learning” of this article is basically a suggestion that solution architects should warm up to idea of tossing tried-and-true integration patterns that treat infrastructure as a commodity that is typically cheaper than bespoke development or operations overhead.

    To me, this read as a pitch of an idea that a DevOps resource spending the proverbial “half-day” to create a custom registration component, as well as adoption of the attitude that consumers can always be made accountable for a) not missing a message and b) being able to consume at the rate that won’t cause the produce to throttle. All in favour of saving storage costs. Because “CLOUD”.

    I’m not sure what environments both the author and the “bozo/critic” character have worked in, but I can tell you that this would be a tough sell in most B2B, or even intra-enterprise integration landscapes I’ve seen.

  7. Message ques are specifically built for the purpose of handling messages in a pub-sub system. Using them usually means a lower development cost and more stable system. I think you have to either process a huge number of messages or have a very expensive service provider if those costs are going to outweigh the former.

  8. Hi Bob,

    Firstly the way “Unknown Attendee who explained above to you” is Awesome. I being a Tech Arch as well convinced with his explanation and the way he looks at is to push the Consumer cost to Consumer itself rather than Publisher bearing the cost for the Consumer inefficiency. Which makes absolute sense for anyone – for example, any banking or private business application, as long as the Producer is ready to bear the costs on behalf of Consumer – for example, Google is a big example for this.

    Thanks for sharing such valuable info with us.


  9. First, I thought this post is about Message Queue in general. But this post is actually about “why using a message queue to facilitate interservice communication sucks”. And the explanation makes sense. The alternative solution proposed is called WebHook and it makes perfect sense to use it in this context. A lot of messaging systems are designed to be used within intranet. It shouldn’t be exposed to outside world. Authentication is one problem but the bigger problem lies in traffic going through WAN.

  10. I sometimes use a hammer on screws and grommets as well. Sometimes a hammer is all you have.

  11. “The place for queues is in the consumer” .
    In my last 15 years I only worked with this setup, and it works perfectly.
    Great flexibility and decoupling.
    Is simply the native way of working with SAP based architectures.