Great, we can control for layout of code, heap, stack and other effects which mess with a performance measurement. However, why do those things have a (statistically significant?) impact in the first place? I guess that hints that with some engineering you could in principle get a speed boost by specifying the layout. "Worst case", you sample randomly and then pick the fastest arrangement, where it is statistically significant.
It could be that the problem arises when trying to measure very small speed increases (small relative differences => noise matters more). But in that case the fact that such a small speed increase is wiped out by random layout effects surely means that time would be better invested in finding a more performant layout?
> why do those things have a (statistically significant?) impact in the first place?
In a word, caches. Not just the instruction / data cache, but also page faults and micro-architectural features like micro-op caches, instruction TLB entries, loop stream buffers, cache-line alignment, and aliasing in the branch predictor tables (which can also be thought of as caches).
I suppose I was musing more along the lines of "why isn't this a solved problem". Clearly, it isn't an easy one or compilers would already take this into account and then the statistical variance would be reduced.
Yes, it's important to measure and control the effects of things like topology/affinity, which can have factors of several effects on HPC program efficiency. You shouldn't suffer much variance on single-node compute, but it's different when you're subject to effects on/of the network fabric and filesystem.
It could be that the problem arises when trying to measure very small speed increases (small relative differences => noise matters more). But in that case the fact that such a small speed increase is wiped out by random layout effects surely means that time would be better invested in finding a more performant layout?