Wondering if it is okay to remove .sim files from the "SavedSims" folder, because having to remove sims from the in-game bin in CAS is quite the headache because the load times are absolutely horrible

Yes. It only affects sims you made and saved in CAS. The only problem is if there are sims you want to keep then you'll want to remove the ones you don't want in CAS.