<Plays game again to try and figure out whether he was just being lame...> I think it is the way the left and right controls work - the fact that even on the static base level moving left and right also moves you "outwards" - where if you just hold down left at the start you move in a slow spiral outwards until you fall off the platform. My brain expects it to move in an exact circle around the totem - even though though that isn't remotely realistic.
Once you're on a totem platform it feels absolutely fine that you're being thrown outwards by the force of the spin - where if you jump straight up and down a couple of times you end up falling off the outer edge of the platform. But when you combine the two factors in a jump to the left or right from one platform to another my brain seems to fall over trying to account for both factors at the same time - the outward motion from the moving platform and the extra outward "kick" from holding left or right.
I guess most of this is down to me having certain expectations from a lifetime of unrealistic game physics rather than anything "wrong" with the game, but yeah, that was the best I could come up with (other than me being lame...)
Did I mention I'm really, really bad at giving short answers? I probably should have said that first...
Oh, and a couple of times on the fourth level I managed to wedge myself between two of the wooden discs :D