Very fair point, but what about the fraction of people who not only consider it an expense, but an affront to moral dignity? For example, Politico did a poll which showed that 46% of Americans believe that everyone should use the bathroom of their birth. Should their comforts be sacrificed for the sake of 0.6% of the population?

If people do things that have no material impact on you, and yet you believe are a "moral affront" then you don't understand where your liberty ends and someone else's begins. Those people need to learn that lesson, and then get over themselves.

You don't get to impose your morality on other people. Morality is a personal belief, and nothing more. It's that simple.

"Believed that they should" is not necessarily the same as "affronted if they don't".

There are, for example, a number of people who would not get an abortion, but believe in the right of others to doso.

But does it matter in that context? Does/should a belief be weighted higher if it causes "offense" - this would mean the easily offended would have their beliefs weighed higher, and thus people would be encouraged to be so.

I don't think a belief should be necessarily weighted higher because it causes offense (or not), no.

I was trying to indicate the difference between "I would never" and "no one should ever" - the former being a belief that attempts to say nothing about what other people should or should not do, where the latter certainly says something about what everyone should do.

Doesn't the former lack the moral principle of universalism?

This issue is separate from whether bathrooms should be unisex