I didn't really notice the stars because I didn't see any indicator of what gives you the stars or how fast exactly you have to be? (I actually thought they were pickups I was missing) Because speed is the only reason to match the shape, if the player isn't interested in stars, the acceleration loss actually becomes a good thing (I've slipped off a ledge to my death many times due to going too fast).
I'd say the scaling might need a purpose other than speeding through the right hole, like if scaling in a direction also scaled your speed in said direction (so y scaling gives more jump height, forward scaling makes you faster forwards, etc.).
I think having to get to a certain location to revert your scaling had some puzzling potential, i.e. you have to get to location A before you can resize yourself and fit into location B rather than a linear route. Though, this wouldn't work in this specific game because having time pressure and puzzling don't mix.
A more brute-force method would be to give the stars a purpose, like an alternate ending if you manage to get a certain amount. It's elegant and easy to implement, but it also forces players to prioritize speed, and time pressure can be rather stressful.
I think the best solution really depends on what you think is most important about the game. Is it about beating the clock and executing moves with the perfect timing? Is it about using the mechanics given to you and the map's features to methodically solve each level? Is it about coming up with new and novel ways to use a mechanic already given to you? As long as it feels rewarding to engage with, I'd say it works.