I havent ran into any issues with having multiple harems, you just have to have some foresight and meet all the conditions for each character in each harem, or a character may walk.
The two to watch out for are the home harem and the pixie harem.
