So there should be an animation for downtime between switches to keep this from happening. While it may seem a bit troubling, it will be effective at patching this glitch and maybe giving the characters a little more personality.
Or disable interact inputs once Yes is selected. Or switch character variables as soon as Yes is selected. An animation is a lot of work- this isn't a big bug.
