Was in the same situation as a math/physics undergrad. I took an ML class with the same curriculum as yours and it was easier for me than most of the CS majors in my class, as it was focused on algorithms rather than implementation. You should be fine!