But I've read that educated Romans associated milk drinking with barbarians, so there'd be a social component to avoiding milk, too, even when it was available. Though I suppose those educated (and probably wealthy) Romans would have plenty of other sources of protein.