I do enjoy the idea, I love falling block type puzzle games, but man... it's just a hair too clunky for me to really sit down with.
I sorta would rather have either the person controls/swap controls be separate, or have the swapper be tied to the player position. That might also solve the problem I have with expecting to be able to swap with the edge of the field.
Also there was a small bug where I restarted too quickly and the music overlapped. Once I failed and restarted again the music reset properly though. Nothing too bad though.