The answers for this will vary widely, but the thing I think many people overlook when planning out expenses is a plan to back up the data. Having the file server is great, but start planning now for what to do when it breaks. Where will backup copies of your data live and how will you restore it?

As to the server itself, the hardware completely depends on your desires. Some like second hand enterprise gear; others prefer purpose-made home NAS devices or a DIY rig. On the software side my thought is keep it simple if you’re starting up. There are good readymade options (TrueNAS, XigmaNAS, openmediavault, unraid, etc). They’re all great and they help get up and running quickly. They also have a lot of tempting knobs to turn that can cause unexpected problems if you don’t fully understand them.

To my mind file servers have to be reliable above all else, so I’d avoid running anything besides file sharing on your server until it’s running like a top and then only add more layers one at a time.

Sorry for all the philosophy, but I really do think this is a common stumbling block for people getting started.