Interesting concept, but my brain kept forgetting which magnet was attached to which button. It might seem strange, but literally placing A and D on them could reduce the barrier to play.
I consistently had the magnets clip through walls and bug out, but I understand that Jam games won't always be bug free. Cheers!