It's actually pretty complicated, but in general, viruses need certain receptors on cells in order to be able to infect them. These receptors are usually specific to certain types of animals, which is why you can't just give a virus to any animal and expect it to be able to infect them. Additionally, even if a virus is able to infect a certain animal, it may not be able to replicate effectively or cause disease, which is why a lot of cross-species transmissions don't result in an outbreak. In short, it's all about the virus's ability to bind with specific receptors on host cells and replicate effectively.