As a 41-yr old hacker (and late-comer to the game; I didn't start until I was well into my 20s) I'd just like to thank you for sticking up for us "old-timers." It really bothers me that people think you're past it when you get past 35 or so. 'Scuse me if I'm not dead yet and have no intention of acting as though I am.
My project is at http://surftrackr.net/ It won't ever make me rich, but (a) that's not the point, and (b) it's my way of making a contribution. While there are programmers out there who can do a better or worse job than I have, I can guarantee the differentiating factor is not their age.
I too, started late in the game. With a brief introduction to Basic in school, and some programming classes in undergrad, I graduated with a Bachelors in Production Engineering, and it was after that I realized that computers (and programming) is where I felt just right. I worked on my first major (Java) project in grad school when I was 21. Haven't looked back since and over the years have learned several languages.
I will admit though, that a)starting late b)not having a formal education in computer sciences is a little frustrating at times. I spend my evenings trying to play catch up with those who had a head start (reading up on compilers, language design, even SICP). Don't get me wrong, I am still as fascinated with computers as I was 8 years ago (I turn 30 today, the 1st actually), but every time I read a brilliant comment on a language or a framework on YC News, I know I have a long way to go before I will have that level of competence and understanding.
To answer your question, I don't think its never too late to pursue your passion. In the long run, when you look back, you would rather see having spent time doing things you loved to do, rather than spending a majority of your life as a zombie, in pursuit of a paycheck than anything else.
From another late starter, I can only wish you the very best.
My project is at http://surftrackr.net/ It won't ever make me rich, but (a) that's not the point, and (b) it's my way of making a contribution. While there are programmers out there who can do a better or worse job than I have, I can guarantee the differentiating factor is not their age.