Ok I played again and as long as I didn't touch any arrow keys while swinging I could make it through. But this time I think I went a different route through City 2 so I didn't hit the part that frustrated me before.
I think the issue happens because I like to hold the up arrow the entire time I'm driving to keep momentum. But the swinging seems to lose momentum when I'm still holding up arrow.
I got to the end, the story is getting wild! I feel like mechanics get introduced really fast, each world could have way more levels to explore the mechanics more. Also now that I have a rocket it feels like the game changed from a trials type game to a rocket booster type game? Like the rocket mechanic overrides all the other mechanics.