It's actually not hard to understand; as society progresses and times advance, the era of one man with multiple wives is long gone. Even the most philandering of men today have only two or three children at most. Moreover, according to Ye Xiu, both his father and grandfather were deeply devoted, loving only their wives. After giving birth to his father, his grandmother damaged her health, so his father was an only child. And his mother herself was ill-equipped for childbirth, but she loved Ye Xiu's father so dearly that she wanted to leave him an heir. Thinking she was doomed to die sooner or later anyway, she preferred to leave him a token of their love.
Thus, a month after giving birth to Ye Xiu, his mother couldn't hold on and passed away. So now, the Ye Family is truly dwindling in numbers; he doesn't even have a choice in the matter.