I completely agreeYeah, and if it does get you down, then rehome a fish. It's sadly sometimes a necessary part of the hobby so that our tanks are relaxing to us and not stressful.
There are very few people on this site who haven't rehomed at least one fish due to aggression. It sucks but it can be a real relief after the fact.
I still think it's worth waiting out a bit with your crew.
Keep us updated!