You are aware that NPM already does that, right? It’s even safer because the network requires immutability so there’s no way to trojan a package without shipping a new version for everyone to see.
The real problem is why I mentioned auditing: the attacks we’ve seen over the years have been updates from maintainers who followed the normal process. Auditing is the most reliable way to catch things like that because the problem isn’t the distribution mechanism but the question of being able to decide whether you can trust the author.
Compiling once from a tarball and reusing that can definitely reduce the number of times you would need to trust something from a third party.