I used to have that problem about half the time, and the other half of the time, the opposite-- where I'd assume they knew nothing about my field, and elide details they were actually interested in. So now I tend to ask lots of probing questions when explaining things, which seems to annoy people who just want me to get to the point! Surely there is another middle ground; the one I'm trying seems to not work.