I looked into the code in more detail and I realized I didn't make turning framerate-independant. Well.. I tried, but I didn't do it properly. I used a mix of Time.deltaTime and Time.fixedDeltaTime in Update(), when I should have only used deltaTime. That's why the turning was incredibly slow on anything less than 60fps. Even 60fps was slow since I tweaked the settings when using 144fps...
It's a pretty dumb mistake, but I already fixed it and will update the game when the voting period ends.