Viewing a single comment thread. View all comments

mayiSLYTHERINyourbed t1_iu3dafc wrote

On a regular basis. We care down to the ms how fast inference or training is. In my last organisation we had to process like 200k images while inferencing. At this point even a delay of 2ms would cost 6.7 minutes just for getting the feature vectors. Which really matters.

3

GPUaccelerated OP t1_iu4ve0v wrote

OK right. That's also a project with immense scale.

I guess the bigger the project, the more inference speed is required. But I've never heard about caring deeply about the ms in training. Mind sharing why that was important in that use case?

1

mayiSLYTHERINyourbed t1_iu7im0x wrote

Our use case was in biometrics, where the test sample would usually range in millions of images which needed to be matched simultaneously. Over here even accumulating 2-3ms over each batch or batch would lead to huge delay.

2