NovaBom8 t1_j5h30af wrote on January 22, 2023 at 11:24 PM Reply to [P] Benchmarking some PyTorch Inference Servers by op_prabhuomkar Very cool, great work!! In the context of running .pt (or any other device-agnostic filetypes), I’m guessing dynamic batching is the reason for Triton’s superior throughout? Permalink 3
NovaBom8 t1_j5h30af wrote
Reply to [P] Benchmarking some PyTorch Inference Servers by op_prabhuomkar
Very cool, great work!!
In the context of running .pt (or any other device-agnostic filetypes), I’m guessing dynamic batching is the reason for Triton’s superior throughout?