Right click to change the focus of the magnet onto another object.
Sorry if that doesn't seem obvious, while testing most levels, it seemed much easier and simpler if I assigned a button to change object rather than doing it based on what was nearer.