pommedeterresautee OP t1_ittyyn3 wrote on October 26, 2022 at 8:32 AM

Reply to comment by sam__izdat in [P] Up to 12X faster GPU inference on Bert, T5 and other transformers with OpenAI Triton kernels by pommedeterresautee

Kepler gen is a bit old, but we may increase hardware support in the future.

First Triton is going through a big rewriting and it's expected that some bugs we had to support older devices will be fixed, of course, nothing 100% sure.

Moreover, we plan to (re)explore cutlass which supports at least Tesla hardware (but they said that their -new- work will only target >= Ampere devices).

sam__izdat t1_ittzava wrote on October 26, 2022 at 8:38 AM

Oh, I absolutely don't expect you folks to support a Tesla from 2014 -- I just kind of crossed my fingers and tried it anyway, on the remote chance that by some miracle it might work, like a magic spell to speed up this slowpoke.