This is such an awesome article that puts into words that I've been contemplating for a while now. I've been trying to explain this exact scenario to various management folks but they just don't get it and always scream why can't you "queue" them up!
Even I was under the impression that queues are meant to solve the problem of overload; but over time I slowly realized that that's not at all what queues are meant for!
Queues are meant to solve following (not exhaustive list) problems.
1. Tight coupling. An example is decoupling "payments" from "order processing". Here, two independently running processes need to message each other and Queue is one way of messaging.
2. Surge handling.
What typically happens is that #2 gets conflated with "increasing the overall throughput of the system". And it does work for a while since in the initial days the real bottleneck won't be reached. And when it does reach; due to marketing campaign and what not; then you have situation described in the article.
tl;dr In a system, if requests arrive at a rate that is greater than its throughput then no queue can save you.
The fascinating thing about scaling systems is that you will end up discovering a new bottleneck every time; e.g., A bottleneck moving from NIC to a switch and to a router and so on!
Even I was under the impression that queues are meant to solve the problem of overload; but over time I slowly realized that that's not at all what queues are meant for!
Queues are meant to solve following (not exhaustive list) problems.
1. Tight coupling. An example is decoupling "payments" from "order processing". Here, two independently running processes need to message each other and Queue is one way of messaging. 2. Surge handling.
What typically happens is that #2 gets conflated with "increasing the overall throughput of the system". And it does work for a while since in the initial days the real bottleneck won't be reached. And when it does reach; due to marketing campaign and what not; then you have situation described in the article.
tl;dr In a system, if requests arrive at a rate that is greater than its throughput then no queue can save you.
The fascinating thing about scaling systems is that you will end up discovering a new bottleneck every time; e.g., A bottleneck moving from NIC to a switch and to a router and so on!