pretty sure that's how it is coded. Advancing map doesn't advance time of day. Dialogue duel does and special duels like Dark Citizen and Quest tools. Dark citizens even has specific time change due to narrative. If you want to advance time of day, you can sleep it off in your house any time of the day.