More than two months in my spare time. To collect the "static" information, and create the Ansible roles and python scripts to collect the "dynamic" info (the results from the benchmarks) and finally compose the final document.
I lost a lot of time waiting for the benchmark tests to finish ;-)
Yeah, I agree, that would be nice, but as I automate the creation of those tables and cut/paste them every time that I run a set of new tests, to do that would require a lot of manual job to me. That's the reason why I don't do that, to keep it manageable.
And thank you very much to share your experience with us. I think that the same provider can offer you a very different experience in different regions. Even AWS is not immune to that, they had datacenters with some major outages in some US regions while in another countries almost none big problems.
That's a very true statement. There are things outside our control (and sometimes in it) that can result in downtime. Like a datacenter downtime event in 2013 where a road construction crew were performing maintenance and ended up cutting their fiber (the location fiber actually was didn't line up with where it was supposed to be on paper). Since then they've built another fiber connection out of the building.
In the end, what matters is how they respond to these kinds of events and issues and what actions they take to prevent those issues in the future. While I can't say for certain what issues were avoided through proactive measures, I will state that Linode has been great with setting up fixes that hopefully prevents problems in the future. DigitalOcean recently had a problem twice with their billing and control panel system going down (in addition to their API being down). They've also taken action to prevent this in the future and have been fairly open about it (one was their SFO datacenter being down if I recall correctly).
My only gripe about Vultr is that their support responses are very basic. Limited to "try again now" and "we've turned off a neighbor". Honestly I was hoping a solution that would work long-term and communication on how they're working to ensure that. DO and Linode focus on relaying that information and I feel really works with me. This is why I'm really fine with buying through them even if they're technically "twice" as expensive as Vultr. That peace of mind is really worth it.
Well, I would like to include more, in fact I would include Ligthsail for sure if they offered a datacenter in Europe. But to start something like this you have to start small, or the costs will bury you pretty son. As a pet and personal project, the way to limit the cost was setting a clear seat of boundaries, like plans under $5 and available in Europe. And being one-man band to make this affordable in terms of time and effort I had to keep it small at the beginning. I worked on this in my spare time for more than two months. If I find a way to sustain the costs, or the providers itself offer me a way to test their machines, I have no problem to expand the comparison. But in the meantime, I share the tools because I knew from the beginning that I only have limited resources. And after all, reproducibility, independence and transparency are the keys IMHO to have a trustful comparison.
I suppose that is the same with another tools and providers. The community size and the work that the providers put on this makes a big difference IMHO. Another reason to share this in the comparison.
Thanks!
I lost a lot of time waiting for the benchmark tests to finish ;-)