Ah, the conversations need a way to cancel them. A click outside or esc or something
Solved it! That was a neat puzzle!