At some point you will exhaust the quantity of people that have a mutually exclusive set of ancestors to your own, thus the child from any union will have some small portion of ancestors that are on both sides of the family tree. Could it be argued that due to the increased likelihood of males being in the category where they are an ancestor to both the mother and father that OP correct?

Lol, OP could be right because incest