Maybe try using a 3rd party remote desktop system with focus on performance instead of the standard vnc server included in mac? Parsec might work for your use case. The disadvantage is these kind of remote desktop system achieve the performance by utilizing lossy codecs instead of lossless compression used by vnc, so you might notice some banding/artefacts depending on your network condition / compression level. The advantage is the hardware encoding/decoding is very fast, even fast enough for gaming.

https://parsec.app/