I don't think the block movements are too fast. Wouldn't this game be trivial otherwise?
Anyway, blocks building sideways doesn't seem to happen frequently enough at first, and since they happened on the side of the screen initially for me, I thought it was a glitch. I wish that that factored more into the game because it's an interesting idea and makes the game feel more like Tetris and not like a Stacker arcade machine.
Though, I feel like how the scoring is done really can change everything when it comes to design. There are some interesting decisions here - the fact that you lose score if blocks pile too high adds depth, and giving the player a choice to stop adds an interesting gamble for the player, but what might have been interesting too is if you got more points for clearing lines when they're higher up. Then you're incentivized to play more dangerously to score higher, which keeps the game away from being like a timing minigame.