I think it is used in Pytorch’s nn.transformerencoder but a lot of people like implementing their own.

We currently officially support Cuda and CPU, although in principle it could be used for other backends too.

I've actually had pretty good success on using torch.compile for some of the stuff that KeOps works well for!

The segregation is that the "ML logic" is moving into Python, but you can still export the model to C++.

Lots of things. You can see a flamegraph here: https://horace.io/img/perf_intro/flamegraph.png (taken from https://horace.io/brrr_intro.html).

Dispatcher is about 1us, but there's a lot of other things that need to go on - inferring dtype, error checking, building the op, allocating output tensors, etc.

> The Python layer brings most of the PyTorch latency.

This actually isn't true - I believe most of the per-operator latency come from C++.

Have you tried out PyTorch 2.0 compilation feature (i.e. torch.compile)? Might help a lot for evolutionary computation stuff.

Generally folks refer to Keras/Lightning as “high level frameworks”. I think it’s reasonable to call Pytorch a low level framework.

Well, you disclosed who you are, but that's pretty much all you did :P

The OP asked a number of questions, and you didn't really answer any of them. You didn't explain what BentoML can offer, you didn't explain how it can speed up inference, you didn't really even explain what BentoML is.

Folks will tolerate "advertising" if it comes in the form of interesting technical content. However, you basically just mentioned your company and provided no technical content, so it's just pure negative value from most people's perspective.

I'm somewhat surprised Inductor performs worse than cudagraphs given Inductor by default should be wrapped behind cudagraphs.

https://www.encyclopedia.com/literature-and-arts/language-linguistics-and-literary-terms/english-vocabulary-d/consist

This site claims that it's an archaic form of "be consistent with".

I thought it was pretty good :)

I mean... it's hard to write efficient matmuls :)

But... recent developments (i.e. CuBLAS and Triton) do allow NN frameworks to write efficient matmuls, so I think you'll start seeing them being used to fuse other operators with them :)

You can already see some of that being done in projects like AITemplate.

I will note one other thing though - fusing operators with matmuls is not as big of a bottleneck in training, this optimization primarily helps in inference.