A lot of it has to do with short-sighted decision making. Attempt to control some visual aspect of the application takes precedence over long-term issues, such as usability or overall architecture (e.g being semantic and searchable).

Also: biased technology evangelism and special interests influencing web standards and popular opinions. These tow are probably the root issues.