The various processing and encoding pathways in our brains have developed over millions of years of natural selection. You could think of human behavior as a "program" where each generation is an iteration on that program. Plasticity is part of that program: ancestors with neural plasticity that benefited them passed on those genes and thus "conserved" learning as part of the program.
This sounds handwavey. I read The Brain that Changes Itself and it essentially gave the opposite impression of what you are saying: the whole point of the brain is plasticity. People can see, balance or hear through nerves wired to their tongues, for example.
It's handwavey because many of the details aren't known, but what is known is complex and can't really be fit into a HN comment. Also I'm condensing what I know from everything I've read about biology. It's not quite accurate to say that the "point" of the brain is plasticity, and we can tell this by looking at organisms with simple brains and examining what their brain is used for.
Worms use their brain to identify and manipulate objects, to taste objects if those objects are identified as food, and to sense aspects of their environment including temperature and humidity. Bees use their brains to see, to fly, to communicate directions, and to coordinate various behaviors with other bees. There is very little plasticity in these simple organisms: a bee that can't communicate will not find a different way to do so, it will simply die.
As animals get more complex we can see that plasticity becomes more important. A bird needs more complex processing to be able to adapt its hunting strategies for different environments. If environmental changes force a bird to move to a new habitat, we can hypothesize that they are smart enough to adapt their hunting strategies to the new habitat.
The evolution of humans has involved many factors that would favor this kind of plasticity: roaming over diverse landscapes, predator spotting, group hunting, tool use, language, and social communities. The resulting large cerebral cortex that we have from these evolutionary trends also gives us the ability to re-wire sensory pathways in the way you pointed out. But when I say "brain" I think of much more than just the cerebral cortex.
Scientists agree that the Cambrian explosion caused an inexorable trend toward complicated nervous systems and I suppose therefore the initial flexible brain.
What I can't see evidence for is any notion that the brain was less flexible then became more flexible over time. Would it be incorrect to assert that it essentially began flexible, as a result of the appearance of nervous systems, and that's its primary property? Of course complexity increased with time. But did the flexibility also increase?
This is a fascinating question, and I don't think I'm qualified to give an authoritative answer to it, but I think the _primary_ property of brains and brain-like neural complexes is that they encode behavioral responses to stimuli.
Similar to how a signal transduction process mediates between molecular receptors on cell membrane and cell behaviors, the neural complex mediates between stimuli and response at the level of the organism.
But at the same time I can see why you are thinking about flexibility as some sort of "primary property" because they seem to be inherently flexible in some way. I'm locating _plasticity_ more narrowly to the cortex part of the animal brain, but perhaps the cortex is just an amplification of some simpler lower-level flexibility. At this point my knowledge of the subject falls short.
Edit: thank you for the interesting discussion, you have sparked me to dig more deeply into these topics!