Sadly I cant fix it now, but I did change the collider on the player as you suggested and it feels far less clunky.
Thank you for the feedback, it really helped.
