I get that logic, but there is a dose at which, over a lifetime, they do not accumulate to a dangerous level.

There is also a dose which, when added or removed, does not make a difference compared to all the other sources.

Edit: One should also take into account that most people naturally stop eating chocolate before the age of 120. Now, question is, what is such a dose and how often do we see people getting dangerous cumulative amounts of heavy metals from chocolate?