Unless there is habitat in those rocks where life exists that we are aware of, then no why would we care?

Remember many of those large rocks can become an earthbound planet killer. There are still many of them in the Kuiper belt that we are not aware of as light is not always reflective off then.

Id say let's worry more about changing the trajectory of them to possibly bite us in the ass later. Develop a safe system to mine, extract, transport and build planetary defense against large projectiles.