Hey r/MachineLearning! I wanted to share a new PyTorch library I've been working on that I think could be really useful for anyone looking to fine-tune large models with LoRA.

The library is based on the LoRA technique (Low-Rank Adaptation). "which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer." (- quote from the paper)

With this library, you can easily apply LoRA to any PyTorch model with just a few lines of code.

One of the benefits of this library is that it's really small - just 100 lines of code. Despite its size, it's quite powerful and has been tested on a variety of different models, including nanoGPT by Karpathy, and stable diffusion.

It also features an easy-to-use interface that allows you to serve multiple LoRA models at the same time!

Comments

You must log in or register to comment.

pyepyepie t1_j9fj0uf wrote on February 21, 2023 at 3:39 PM

WOW - what a cool idea (the paper), I was not aware it exists! Thank you so much for the simple implementation and the info.

brucebay t1_j9g8al3 wrote on February 21, 2023 at 6:23 PM

Thank you for this. I never used lora except part of stable diffusion training. You linked MS lora lib too. What are the differences between yours and theirs?

cccntu OP t1_j9hsouu wrote on February 22, 2023 at 1:48 AM

Theirs requires you to rewrite the whole model and replace every layer you want to apply LoRA to with the LoRA counterpart, or use monky-patching.Mine utilizes PyTorch parametrizations to inject the LoRA logic to existing models. If your model has nn.Linear, you can call add_lora(model) to add LoRA to all the linear layers. And it's not limited to Linear, you can see how I extended it to Embedding, Conv2d in a couple lines of code. https://github.com/cccntu/minLoRA/blob/main/minlora/model.py

mdda t1_j9s8ptw wrote on February 24, 2023 at 4:36 AM

FWIW, I gave a shout out to minLoRA at our Machine Learning MeetUp (in Singapore) last night : https://redcatlabs.com/2023-02-23_MLSG_Frameworks/#/15/2

cccntu OP t1_j9tfr5r wrote on February 24, 2023 at 1:03 PM

Thanks for the shout out!

brucebay t1_j9i2kd6 wrote on February 22, 2023 at 3:01 AM

Thank you for this clear explanation.

thecuteturtle t1_j9nwdfm wrote on February 23, 2023 at 8:58 AM

Man i feel so behind that i didnt even know LoRA.

JClub t1_j9j5cv4 wrote on February 22, 2023 at 10:03 AM

Sorry, but why do we need another package? Can't you build on top of https://github.com/huggingface/peft ?

cccntu OP t1_j9jz6ov wrote on February 22, 2023 at 2:51 PM

This project started out as me exploring if PyTorch parametrizations could be used to do LoRA, and it turned out perfect for this task! And I simply wanted to share that.
I think it would be interesting to see it integrated into PEFT, too. Although they already have their own LoRA implementation there.