I can't say that I've been in extreme cold or dressed with lots of clothes, but I always undress in order to prevent (too much) sweat. Then I dress back as soon as I'm getting cold.
The best defense against the cold I find, up until about 10F or lower, is several layers of long sleeve tshirts. You can add or remove layers rapidly with a rather good resolution (You can easily have 5 "settings") and if you get wet they are very quick to dry (separate them to dry them fast.) If you are out for several days, you can start to cycle which shirt is closest to your body, to delay discomfort caused by dirty clothing.
