To clarify: it was indeed the outdoor maps; the indoor maps are always the same place you entered from (though the southern walls can get in the way).
As for the bug, I triggered it by interacting with a rake in one of the houses that triggered Blobby's bad memory. I was already at a low sanity level (maybe less than 50?) so maybe there's some phantom-negative number nonsense going on.