

Synology is great because of the easy backup. But it is limiting your ideas for your new NAS. Leave it in “backup corner” :)
Play with ZFS, see how much flexibility it really has, and when you got that up, then write your first own smb.conf - and the new NAS is done.
For the host, proxmox. There’s nothing else at the moment that makes more sense.
Do you absolutely need to put ONE tag on it all and say this is it?