The controls are challenging to get used to, but definitely not broken. Mouse buttons make you reach your arms out in that direction and A and D makes you grip onto wherever your hands are. Ideally you'd wanna reach until your hands are on a coloured hold and grip onto that. Then pull yourself in by moving your mouse and repeat.
I actually played this for the dev, and demonstrated that they were broken for me — at one point, even when gripping a coloured hold, it caused my player character to slide horizontally, out of reach of any of the holds that I had to Reset Timer. So yeah, unfortunately, this has been confirmed, but there's no easy fix.