First, it can leave particles on the way down to the stomach. Second, many molds (and fungus) release toxins, when broken down, which is what is dangerous. This isn't the case of every mold or fungus though, the fungus, penicillin is is the same type of fungus found on your "moldy" bread, there are many strains of fungus and its mostly the breaking down part inside of us that's the bad part, not the fungus itself. (Not always the case of course).

There are also plenty of virus and bacteria which are resistant to stomach acid for long enough to make it through to the rest of the body.