Magnetic fields polarise light. We would see it with ease. They also have no effect on most elements. There are a lot of reasons this is not the case, including observing the bending of spacetime caused by the dark matter. That is down to gravity.