I think speeding up the ball would indeed make it much harder... The thing that bothered me was not that it felt "floaty" but rather that the ball slows down a lot on obstacles, when I would much rather it slide through tubes like a pinball. When it stops halfway through a tube it can feel awkward trying to catch it.
Slowing down the ball when the paddle is running is genius, and for some reason I didn't really notice it. I just knew that it was hard to catch the ball sometimes but not too unfair.
Also if I may ask, is the ball's collider not a perfect circle??