If you’re already typing in the full address for the server every time, since you don’t have an ssh config file, then is it really so much more work to additionally type the port?
Yes because I will forget to type -p 5223 at some point. If you set up a non standard port against my recommendation and I get banned for trying 22, you better be on call 24/7 for free to unban me.