I doubt you've actually been to burning man. If you have actually been, I don't think you spent enough time looking around or talking to people.
People that shouldn't be there (financially) still find a way to go; it's no different than many of the diversions we pursue in life.
How much do you think it costs exactly? Historically, as little as a few hundred could get you there. Less than the air fare many people spend to fly home for the holidays.
The definition of "poor" is subjective; I know people that have gone to burning man that met the federal definition of "poverty". Is that "poor" enough for you?
You also seem to be ignoring the people that can't afford to go that are sponsored by others as a gift.