Without going into too much detail, the odds of something bad go down exponentially, specifically with 1/2 for every extra link in the family tree away from a shared ancestor. 1/8, 1/16, and 1/32 odds per gene (3,4, & 5 links, respectively) are pretty bad across >20,000 genes, but it doesn't take a lot of distance (14 links) before your risk drops to 1/16384, at which point it's basically trivial. By the time you're 3rd or 4th cousins, you're basically genetically unrelated.