If you’re going this route, one trick I use: start the hammer drill on drill and scuff the stone at the whole a bit, eventually switch to hammerdrill and keep your finger on the trigger as light as possible. Those two combined will increase your odds of not skipping the bit and marring the new brick