The larger asteroids do affect your ship more, however that doesen't translate directly into a bigger change in angle of the trajectory. As if an asteroid is larger, you also gain more speed and thus more inertia, making your trajectory more straight. The way you use the larger asteroids is by making the small at first and then increasing their size in order to dramatically change your direction without increasing speed too much.