I don't agree with the premise of this question.

Anti-semitism IS a brand racism. Your question assumes that some racism is more okay than others. No brand of racism should be more acceptable than another brand. Race is not just color of skin. That is a common misconception though.

If you are asking why anti-semitism is frowned upon, it is because they have been killed, enslaved, exterminated, oppressed, etc. at scale for centuries. There are fewer than 15 million Jewish people alive today.