As long as you have ideas and thoughts to express through the game in new levels, puzzles, and visual storytelling, let the game grow. Making a game longer for the sake of length itself isn't as important. Your method of developing under constant feedback from the game is lovely, the sooner you see what your code is doing the sooner you can improve it, or make sure it works properly. Making music is something everyone will have struggles with, and making a long loop of music that has enjoyment and quality to it is not something to scoff at.
How full of a release are you planning for? If you presented the game in its current form to a game designer, what do you think they would say about it?