I think all you need is build-essentials and python-dev.

Matplotlib depends on freetype, libpng, and libjpeg as well. Numpy can be built without any external dependencies, but it will be a very slow installation. You're better off with an accelerated BLAS library (e.g. ATLAS/MKL/etc) and some sort of LAPACK library. Some of numpy's functionality also needs a fortran compiler to build. At any rate, there are other system dependencies beyond a basic compiler and the header files for python.

Ah I think the reason why I didn't think I needed BLAS/LAPACK is because I install R on the server first which installs those as shared libraries.

Nope, numpy needs libblas and scipy needs libatlas, both of which have to installed from the system package manager. I slap my forehead every time I move to a new system and find that the pip install in the virtualenv failed.