This does happen! Sometimes! Small populations or small portions of larger populations will begin looking like their inedible counterparts over time. The problem with this - inedibility isn't a perfect defense. A few inedible individuals will still get eaten and spat out. If the markers that signify inedibility become less reliable (as in, there are greater numbers of edible mimics), then the selection pressures of "don't eat the red striped caterpillars" will weaken, and the selection pressure to be a mimic will as a consequence also weaken. Like parasite - host population trends, this leads to an oscillation between a greater number of mimics and their relative collapse, depending on the specifics of the situation.