I used to just go for it and dry nail it, but my husband introduced me to the nuance of pre-drilling with a small bit.

I have rock lath and plaster. I’ve never heard of anyone having an issue with it being weak or crumbling, the only thing I can think of is possible long term unmitigated moisture in your walls. I raised two sons who could not dent or chip the walls if they tried. When renovating and removing a wall, the sledgehammer bounced back at my (then new) husband who did not believe me when I told him what the walls were made of and how impenetrable it is and it left barely a ding. Every original house on my road is made with rock lath and plaster and my neighbors all have the same experience. I’ve hung curtains and blinds and pictures for 28 years and never had an issue. I would advise you to call a licensed plasterer to find out why this is happening.