Looking back at the board... it wasn't a natural Chance Time space, it was a Red Space that was converted with the Last 5 Rounds Event.
What I think might be happening is that after the duel, the game moves it to the next player (P4), which is just trying to roll to move forward, even though the Chance Time event still occurs. That's probably why changing P3 from CPU to Controller 1 didn't work... or why the left/right didn't (thinking it's just rolling a movement die instead).
I'll let you know if it happens again and check things more extensively.