Virus researcher here. The answers provided here cover most of it - the compatibility between virus and host largely determines the success of a virus jumping into a new species. What hasn't been mentioned much is opportunity. There's probably many viruses like SARS-CoV-2 out there in bats and other animals that haven't emerged in humans (yet) because the circumstances that put them in contact with us haven't happened yet. Once they do we'll be at risk of another pandemic. Likewise, many animals are likely susceptible to some human viruses but rarely get a chance to catch them from us.