I think the issue is that one shouldn't go too far in the direction of skeumorphism or too far away from it. Most people are comfortable with some level of skeumorphism because it can be a powerful usability enhancement. As I type this I'm looking at the "add comment" button below here on Hacker News, and wouldn't you know it, it's got a slight gradient giving it a bevel, suggesting it's occupying 3-dimensional space and can be pushed like a button.
Of course this can be taken too far. When too many skeumorphic accents are added to a design it can cause it to be rigid and noisy. As the OP mentions, if you go too far from skeumorphism you run contrary to how the human brain works. My favorite user interface designs usually have a very tasteful and well placed set of skeumorphic elements with an overall minimalist design. Tactile, not tacky.
Of course this can be taken too far. When too many skeumorphic accents are added to a design it can cause it to be rigid and noisy. As the OP mentions, if you go too far from skeumorphism you run contrary to how the human brain works. My favorite user interface designs usually have a very tasteful and well placed set of skeumorphic elements with an overall minimalist design. Tactile, not tacky.