I think it's based on misconceptions. Social media is light and bite sized so you can just put it down any time and scroll while paying attention to the social situation, plus you might be taking care of important things or work messages. When you open a book you get absorbed into hundreds of pages and ignore people without any convenient excuse.

Honestly I also just have some ebooks on my phone and read those, it's not like people can tell the difference between my reader app and reddit unless they stare at it rudely, and I pay the same amount of attention to people around.