I was just talking about this with a YC applicant yesterday. Scaling up HUGE is a really interesting topic, and a lot of nerds love to talk about it...but the number of sites that actually require extreme scaling is very low. When we first started selling Virtualmin we had a lot of early adopters asking about database and web app replication, load balancing, etc. Not because the folks asking actually had sites that needed that kind of performance, but because they're cool technologies to play with. A thousand paying customers later and the demand for those features has dropped to background noise (the same early adopter folks who mostly just want to play with it rather than have the traffic to justify it). Far more of our users are asking for the ability to run more (more sites, users, mailboxes, applications, etc. and not generally more reqs/second) on a single server rather than the ability to spread load across many servers.
The performance of hardware has managed to keep pace with the needs of the vast majority of websites over the years, to the point where very few sites (like the top 500 or so) actually ever need more than just the basic scaling ideas that are easy for just about any sysadmin to implement (split mail, web, DNS, and database onto independent boxes, use memcached, maybe a web load balancer like Squid or pen, etc.).
Could it be that the people who select Virtualmin aren't the same people who need to scale up? Not a rhetorical question, I don't know you guys well enough to have an idea.
> The performance of hardware has managed to keep pace with the needs of the vast majority of websites
That makes sense, but the absolute number of web sites that need to scale up in a big way is also growing, even if it could actually be shrinking as a percentage.
It could very well be a bit of a situation where, "We don't offer a scaling solution, so customers that need to scale don't come our way, so we don't hear from customers that they need to scale."
But, it's worth noting that my previous (now defunct) company was entirely devoted to web performance and scalability. It's not a volume business--the folks who need it spend a lot on the problem, but their just aren't that many who need the extreme solutions. I've often chuckled when the few Virtualmin customers who do want scalability have explained their requirements and they match to a great degree the products I was building five years ago (and found to be a niche that wasn't worth continuing to expend effort on). If I thought it would be profitable to pursue, I could certainly revive some of those products in the context of Virtualmin...but I think there are far more profitable areas for us to work on.
Everything related to the web is growing, so all ships are rising, including scalability issues...but I believe several others are rising much faster.
The performance of hardware has managed to keep pace with the needs of the vast majority of websites over the years, to the point where very few sites (like the top 500 or so) actually ever need more than just the basic scaling ideas that are easy for just about any sysadmin to implement (split mail, web, DNS, and database onto independent boxes, use memcached, maybe a web load balancer like Squid or pen, etc.).