I remember when Brackeys made Diggin' he said that the time limit felt more like a constraint than something to push you. There is only so many points you can get - the skill ceiling is low. What he did is he added time pickups, which you could collect to get more time. I think in your game you should lose even more time when touching a cube of a different color, and get time when you destroy a cube. That way, you could have a good main game loop.