I had this issue, turns out you *SPOILERS* need to select "You visit there often" or else you won't understand how unusual it is that they're not there.
