Personally I didn't find the start too bad. I think it might be a good idea to make smaller, lighter asteroid which are a bit easier to move but don't give you much. That way you can choose to make multiple slightly easier trips or to struggle with the big bulky ones that would reward you more.
My main issue was the collision boxes only being on the ship and the engines of the hauler module. If the scaffolding also collided with the asteroids, acting as basically a giant bucket on the front, then I think it would be much more fun. Just putting those collision boxes on a different layer would allow it to collide with the asteroid, but not the ship. You could still lose it if you were going too fast and tried to just stop without rotating, but it wouldn't fall out the back between the engines that way.