Running on a privileged port doesn't change which other hosts can connect to the privileged server though. At best it tells the client that the server is probably running as root (or has the appropriate capability bit).

If you're worried about an impostor sshd on that host then I would tend to agree that it's simply not prudent to be connecting to that server at all, let alone passing key material or credentials.