I find having a "Harem" is having multiple girls as your wives/concubines/servants to basically use as you wish (with you as a sort of boss), whilst polygamy is having multiple girlfriends/wives in a more romantic setting where you don't use them but actually have feelings for them.