The problem is that Columbia is situated so far away from the other two, and from the "heart" of the city, that the university simply cannot have the same overarching influence on society that other schools that are better integrated with their surroundings do (like Harvard and MIT with Cambridge).
In addition, it's a matter of pure density- Cambridge and Boston, to a lesser extent, is seen as a college town because of the density of college students living in that area. In New York, college students compose such a tiny percentage of the overall population, and exert such a small influence on everyone else, that they are hardly given a second thought when determining the dominant themes in a city's ideology.