You're 100% right about it being slow; I almost submitted the game with just the expert map, which would have been even slower!
Ultimately there has to be enough time for the player to see whether it's going to be a click or a flag, and then enough time to move the pointer out of the way if necessary. I had some ideas about streamlining the game a bit to only have a few key moments where the player had to bump the pointer, but had to keep it simple for the sake of time.