Just tearing down some drywall isn't likely to release much asbestos, even if it has it. They didn't put a lot in the boards or mud, it was the workers who were sanding the stuff for 8 hours a day that were in trouble.

Frankly I wouldn't have gotten the test done at all. For a low-asbestos product, it doesn't change how you deal with it (ventilation, don't grind or cut, breathing protection) and if it comes back positive, you're going to have to report that finding to anyone you sell the house to.

But yeah, it takes multiple decades to cause cancer, is almost always because of chronic exposure, and doesn't hurt children in particular like lead. So maybe take it easy on the guy.