I think I only managed it twice to get to the worm in about a dozen attempts, I just don't get the wall jumping mechanic, not only because of the direction but the timing required is really tight and you can't grab a hold, take a breath (so to speak), and get ready for the next jump.
Moving and jumping worked with a gamepad (PS3 controller), but I couldn't find a button that would have done the possessing of the worm...