The reason HTTP is trivial to scale and databases aren't is that the HTTP daemons are largely stateless, while databases are all about managing state. Doing that in a scalable, reliable, consistent way is just a fundamentally hard problem. Oracle and DB2 haven't done particularly well at trying to solve this (within the constraints of a traditional RDBMS), let alone the various open source projects.
I spent a while trying to build a synchronous multimaster replication system for Postgres. I think we made two main mistakes:
1. Trying to provide the traditional ACID semantics that people expect on a single-site DBMS isn't feasible, at least without incurring a very significant performance and complexity overhead.
2. Horizontal partitioning is key. If you make it easy for the user to partition their data, you now only need to maintain consistency over a single partition.
I spent a while trying to build a synchronous multimaster replication system for Postgres. I think we made two main mistakes:
1. Trying to provide the traditional ACID semantics that people expect on a single-site DBMS isn't feasible, at least without incurring a very significant performance and complexity overhead.
2. Horizontal partitioning is key. If you make it easy for the user to partition their data, you now only need to maintain consistency over a single partition.