I like the idea of dragging the UI as a gameplay mechanic but you could do more with this beyond movement etc. Perhaps you have a more traditional UI that's also diegetic. You can think about 4th wall breaking with this style of game, too.
In terms of your actual controls, it feels like the controllers move too fast for the player to keep up with. As having to drag the controls onto the character is clunky and takes time. This can be intentional though to make sure that the game's speed is restricted by the player's own dexterity (physical skill check!), but be sure to balance the rest of the game around it