The major advantage IoC provides is decoupling of components. Components do not instantiate their dependencies and therefore are not coupled to specific dependency implementations. Any dependency that satisfies the interface the component expects can be injected.
This is especially useful when implementing tests because it is easy to inject mock implementations of dependencies.
You have it a bit backward IMO. Java (specifically) unit tests normally require that you indirect your dependent code through an object reference in order to mock the implementation, thus creating a composition problem. Unit testing in Java turns a control flow problem into a data flow problem. You then need help in getting all the data to the right places, a problem that unit testing created. IoC containers are a solution to this problem; they then move part of the implementation of your program into your config, or embed it implicitly in constraints in how things are composed. This is usually not an improvement in ease of understanding or navigating a project, since it optimizes for changing it.
Personally I'm deeply suspicious of mocking as an approach to testing code. I much prefer a functional approach with explicit data flow between parts of the system, rather than indirecting control flow through vtables using complex ad-hoc protocols that become part of your specification by getting baked into mocks in tests. IoC and mocking promotes dataless Verber, Doer and Handler classes that have no real place in an OO model of a system.
We shouldn't forget that decoupling isn't good for its own sake. Things that change together are already coupled; decoupling them at the implementation level is usually an obstacle in every way, whether to understanding, effort, performance, whatever. Abstraction use vs programmer experience generally follows a hump shaped curve; beginners know nothing, intermediates abstract everything, while experts use fewer more powerful, more composable abstractions and code more directly elsewhere. Mocking IMO optimizes for the middle case of too many non-reusable abstractions.
Part of the problem is Java environment itself, other platforms allows exporting single instances (import an actual singleton), so there is not so much need for DI
Also, you don't need to swap everything, your application (in fact) has hard dependencies. And mocking everything is not always the best way to test system (sometimes it's totally okay to mock the whole db)
This is especially useful when implementing tests because it is easy to inject mock implementations of dependencies.