Nvm I fixed it by...starting NG+, going to camp to do an animation and then reloading a save back before I did NG+?? I don't know why that fixed it but okay then.
I think the issue is caused by sneaking into princess castles, exhausting your endurance and then going into camp and starting a scene. Not completely sure but that's around when it started happening.