Interesting idea and very challenging.
I think something that could balance out the difficulty would be to aim with the mouse or the left controller stick instead of 'A' and 'D'. You could still limit the steering speed of the main character if you're worried about that. As it is, it can be pretty difficult to juggle both the player position and the color choice.
Regardless, good job overall :)