Your reasoning about pause makes sense. Maybe you could have a "casual" mode that allows pause, and a normal/speedrun mode that does not allow pause. Or just leave it as-is - the lack of pause isn't a deal breaker!
