We have customers for whom prediction time is critical
As long as we meet the SLA though they're happy. So it's a balance of cost and required level of performance. If the increased speed can pay for itself then they will pay but at the moment it's pretty much faster than human perception.
When I worked in algo trading though speed was everything but there you're looking in router FPGAs for fastest possible speed.
_DarthBob_ t1_iu4jm4y wrote
Reply to [D] Do companies actually care about their model's training/inference speed? by GPUaccelerated
We have customers for whom prediction time is critical
As long as we meet the SLA though they're happy. So it's a balance of cost and required level of performance. If the increased speed can pay for itself then they will pay but at the moment it's pretty much faster than human perception.
When I worked in algo trading though speed was everything but there you're looking in router FPGAs for fastest possible speed.