When our main character asks victor why they didn't want to let her inside the house . Apart from replying that they thought she was a ghost, he also said he was afraid she would lose hope. According to me I think he was implying that he and his brothers had already lost hope of returning home thus the reason they could no longer hear the voices of their loved onces.