It’s likely waiting for you to look at the screen on the times it takes longer. See if you have require attention for face id enabled in the settings. I'd keep it on, but you can turn it off if it annoys you.