People were shocked , visibly, at my current job when I told them I didn't really do a lot of programming in my spare time. I made sure to not mention it anymore to someone I don't know / trust very well there, they looked like I had breached some kind of unwritten rule or requirement to work for that company - an unvoiced "You have no passion for your job and you do not belong here", of sorts.

That wouldn't be something they could fire me over though, what I do in my spare time is my business, and we have laws that protect that kinda thing.