I know this likely won't help if there's something that gets in its way, but if you hold shift before you grab something, and continue to hold shift, it'll activate a push/pull mode for more control. (Shift being whatever your sprint button is, so keep that in mind in case you changed it.)