If you truly think your cat will be happier with your ex, that seems like the right decision. From what you describe your cat is used to having plenty of fellow cats to hang out with. It may be quite a shock to move them.

However, if you keep your cat's best interest at heart, nobody would blame you for keeping YOUR cat.

Does your ex have a claim to this cat, or has it always been exclusively yours?