If your trying to make the harem with everyone possible then whenever asked to visit or talk to a character just do it. Not to sure about specific people but I would guess just don't talk or visit them, i don't know the consequences cause I'm doing the harem route.
