That's a very classist remark. It implies that having been poor completely nullifies the value of my opinions.

I mentioned being homeless as shorthand for "I used to go to homeless services, so when I say that most of them suck, that is firsthand testimony." I have also had a college class on homelessness and public policy, I have six years of college, yadda.