> That said, even returning the rays to parallel does not mean recombining them. At the first surface they are given different angles, splitting one beam. Returning this set of beams to parallel just means that the colors do not spread even further once they've left.

This is true, but the effect is pretty small, much narrower than a typical light beam, and dominated by diffraction, so you just get fringes of very slight color.