Our use case was in biometrics, where the test sample would usually range in millions of images which needed to be matched simultaneously. Over here even accumulating 2-3ms over each batch or batch would lead to huge delay.

On a regular basis. We care down to the ms how fast inference or training is. In my last organisation we had to process like 200k images while inferencing. At this point even a delay of 2ms would cost 6.7 minutes just for getting the feature vectors. Which really matters.