[D] GPU-enabled scikit-learn

Submitted by Realistic-Bed2658 t3_zzhg84 in MachineLearning

Hi everyone. I was trying to find a possible gpu-enabled version of scikit-learn, but I was surprised to see that most of the libraries are written for NVIDIA GPUs. While nvidia GPUs are very common, I do feel that Apple’s gpu availability and the popularity of AMD demand a more thorough coverage.

I was wondering whether coding scikit-learn on top of pytorch with GPU support could be something the community would be interested in.

I do not work for Meta. I would just like to do something useful for the community.

Cheers.

33

Comments

You must log in or register to comment.

miss_minutes t1_j2bph94 wrote

https://stackoverflow.com/a/41568439

https://scikit-learn.org/stable/faq.html#will-you-add-gpu-support

Scikit learn isn't meant to be used with the GPU. Machine learning that's not deep learning don't benefit as much from GPU compute. Sklearn also uses optimised native libraries when possible (e.g. libsvm and liblinear), so if you were to implement the sklearn API on top of pytorch, it's very unlikely you'll beat the performance of sklearn.

58

noobgolang t1_j2bucr4 wrote

Well

  1. Most scikit learn will not leverage gpu
  2. Yes fuck nvidia
0

Realistic-Bed2658 OP t1_j2by65z wrote

Replying to miss_minutes (#1,206,218)

thanks for the links, but I disagree for the most part.

DBSCAN and LOF most likely would benefit. Even their own MLP model would inherently benefit from it ( I do believe somebody willing to train a neural network most likely would use PyTorch of TF).

Also, the fact that today non-DL ML is mainly CPU-based doesn’t mean that in 5 years from now this won’t change. Personal opinion here, though.

−5

fcharras t1_j2c2ec6 wrote

It's not available yet, but there is exploratory work being done by scikit-learn devs on the matter. I'm a dev being specifically tasked with this work myself.

The path to a possibly GPU-enabled scikit-learn that is being explored currently is by having a plugin-based system that would enable users to leverage performances from accelerators including gpu-accelerated computing. The hardware-specific optimizations would be maintained and distributed in separate plugins rather than in the main scikit-learn package.

Many people seem to be interested by this idea, including people at Nvidia (for cuda-enabled plugins) and Intel (for optimizations targetting intel GPUs). I'm myself more specifically tasked at using a SYCL-based software stack, which for now is mostly designed with compatibility with intel hardware in mind but has potential of interoperability with Nvidia and AMD hardware at its core.

More about this:

I wouldn't recommend trying to use those experimental branches yet but the 2023 year might see announcements and releases related to this if the project is successful. In the meantime, early users, feedback, and contributions would be very welcome.

6

geoffh2016 t1_j2c3bbf wrote

Isn’t that skortch?

Maybe you want to accelerate other parts of sklearn, based on your comments, but I’d probably ask the project about adding those features to skortch. The NN at the least is already there and IMHO pretty good (at least to help my students migrate from sklearn to PyTorch while they learn).

20

Longjumping_Essay498 t1_j2c3wlo wrote

They use parallel processing and threads that's sufficient for ml not so deep

2

chatterbox272 t1_j2c9gnj wrote

>I do feel that Apple’s gpu availability and the popularity of AMD demand a more thorough coverage.

Apple's GPUs are 2 years old, and although you didn't mention them Intel's dGPUs are <1 year old. Both account for a relatively small portion of users and an effectively zero percent of deployment/production.

Most non-deep ML techniques aren't built on a crapload of matmuladd operations, which is what GPUs are good at and why we use them for DL. So relatively few components of sklearn would benefit from it and I'd be deeply surprised if those parts weren't already implemented for accelerators in other libraries (or transformable via hummingbird). Contributing to those projects would be more valuable than another reimplementation, lest you fall into the 15 standards problem

10

AerysSk t1_j2cgmo4 wrote

Replying to Realistic-Bed2658 (#1,208,518)

If you are looking for a GPU version of scikit-learn, I think Nvidia is making one, and they call it cuml. Note that, not all algorithms are implemented, and there will also be some missing functions as well.

However, a note about Apple and AMD GPU thing: they are on the rise, but not until a few years later that they will become usable. My lab has only Nvidia GPUs but we already have a lot of headache dealing with Nvidia drivers and libraries. At least for a few years, we do not see any plan switching to AMD or Apple.

11

jeanfeydy t1_j2cnd36 wrote

Replying to miss_minutes (#1,206,218)

From direct discussions with the sklearn team, note that this may change relatively soon: a GPU engineer funded by Intel was recently added to the core development team. Last time I met with the team in person (6 months ago), the project was to factor some of the most GPU friendly computations out of the sklearn code base, such as K-Nearest Neighbor search or kernel-related computations, and to document an internal API to let external developers easily develop accelerated backends. As shown by e.g. our KeOps library, GPUs are extremely well suited to classical ML and sklearn is the perfect platform to let users fully take advantage of their hardware. Let’s hope that OP’s question will become redundant at some point in 2023-24 :-)

30

rlew631 t1_j2ct0zq wrote

Nvidia rapids was pretty decent when I tried it out

3

username4kd t1_j2d39xv wrote

Nvidia RAPIDS suite is worth looking into. Specifically I think cuML is supposed to be mostly the same API as sklearn

3

Top_Ad6168 t1_j2e84wc wrote

If you are interested in just predictions you can try Hummingbird. It is part of the PyTorch ecosystem. We get already trained scikit-learn models and translate them into PyTorch models. From them you can run your model on any hardware supported by PyTorch, export it into TVM, ONNX, etc. Performance on hardware acceleration is quite good (orders of magnitude better than scikit-learn is some cases).

If you are interested in trying to do the same thing for training, let's open an issue and try to work through it for few simple models. We have some work on fine tuning pre-trained tree-ensemble models using PyTorch, but this is a bit different..

Paper: https://www.usenix.org/system/files/osdi20-nakandala.pdf

Fine tuning paper: http://www.vldb.org/pvldb/vol15/p11-yu.pdf

Paper on doing the same for other type of computations (e.g., graph algorithms): http://www.vldb.org/pvldb/vol14/p1797-koutsoukos.pdf

Paper on doing the same for SQL queries: https://www.vldb.org/pvldb/vol15/p2811-he.pdf

1