It was more so that before you had the concept that one child had to kill all his brothers. That did kind of balance it out.
The real downside of the harem system was that it made the royal family incredibly insular. Unlike the Europeans who were constantly mixing.
Also it's important to keep in mind that harem concubines were slaves. That's why they were chosen as they had no rights of their own.