Benjamin Bossan retweeted
evil qwen got us playing with neurons in peft/transformers, and I got to say I really feel like a brain surgeon
tweak one neuron AND THE MODEL STARTS SCREAMING IN ALL CAPS
tweak another one and it behaves like an oracle; apply more pressure, it talks like a god.
tweak both?
Happy to announce the newest 🤗 PEFT release, v0.21.0 🎉
github.com/huggingface/peft/…
Check the thread below for a few highlights of this release.
ALT https://github.com/huggingface/peft/releases/tag/v0.21.0
ShadowPEFT trains a small shadow network (which can be a pretrained model) alongside the frozen base model. It’s a good choice when you care about downstream or OOD performance or when you want the small model on the edge and the frozen base in the cloud. Added by @xmlee97.
There is of course much more, including performance improvements, better handling and checks of multiple adapters, etc.
Again, check the release notes for details on all of this and let us know what you think.
github.com/huggingface/peft/…
PEFT v0.20.0 was just released and as always, it contains a ton of new additions that are worth checking out 🤗.
Besides many new PEFT methods, we added new image benchmarks to compare the methods in a fair manner and we made the docs more useful. Check the thread for details.
Finally, to make it easier for you to find the best PEFT method for *your use case*, we created the "PEFT Shop". This @Gradio space allows you to browse the PEFT methods according to their metrics and capabilities and to easily compare them:
huggingface.co/spaces/peft-i…
Of course, we also added a lot of new PEFT methods:
- HiRA
- GLoRA
- BEFT
- MonteCLoRA
- VeLoRA
- Uni-LoRA
- FRoD
- MiCA
- DEFT
- KappaTune
Check out the release notes for more details: github.com/huggingface/peft/…
When it comes to fine-tuning, LoRA is dominating. But is it the best option? We investigate this in our latest blog post on HF.
You'll learn which PEFT techniques work best on our benchmarks and how to choose the PEFT technique that is best for your use case.
Check out our blog post "Beyond LoRA: Can you beat the most popular fine-tuning technique?"
huggingface.co/blog/peft-bey…
Today, we released PEFT v0.19.0 and it's a big one. Not only did we add 9 new PEFT methods, the release also contains a bunch of improvements to make PEFT more useful. Check the thread for details:
We also added 9 new PEFT methods. There is not enough room here to describe them in detail, but here is a list:
- GraLoRA by github.com/yeonjoon-jung01
- BD-LoRA by github.com/Conzel
- Cartridges and TinyLoRA by @krasul
- PVeRA by @leofillioux
- PSOFT by github.com/fei407
- Lily and PEANut by @tmux_1
- AdaMSS by github.com/LonglongaaaGo
Find a short description of these PEFT methods and the full release notes here: github.com/huggingface/peft/…
