While you have a point, I would advice against this. CS is fundamentally more project focused. You really need the experience of programming assignments and course projects which can be quite a lot. Then there is the graduation design projects on top.

Math/physics graduates really struggle to keep up with the pace or even working in a group.