A very simple but very cool idea! Haven't seen anything like this on the jam so far. With some polish it would make for a great time wasting game.
Maybe the speed ramps up a bit too fast, could start a bit slower and slowly build out from there. More difficult levels could even include more dice to keep track of.
All in all, a really cool and original concept!