I dont see in Big Sur any system-wide option to close an app after closing the last window.

There is only the opposite: "Set all app windows to close when quitting apps"

I currently use a 3rd party app to close for me apps when I close all of their windows.