Adapters v1.1.0

[calpt]

This version is built for Hugging Face Transformers v4.47.x.

New

Add AdapterPlus adapters (@julian-fong via #746, #775):

AdapterPlus (Steitz & Roth, 2024) is a new bottleneck adapter variant optimized for vision transformers. Check out our notebook for training AdapterPlus adapters for ViT models.

Easy saving, loading and pushing of full adapter compositions to the Hub (@calpt via #771):

The newly added save_adapter_setup(), load_adapter_setup() and push_adapter_setup_to_hub() methods allow saving, loading and uploading complex adapter compositions - including AdapterFusion setups - with one line of code. Read our documentation for more.

Enabling full gradient checkpointing support with adapters (@lenglaender via #759)

Gradient checkpointing is a technique for enabling fine-tuning in very memory-limited setttings that nicely complements efficient adapters. It is now supported across all integrated adapter methods. Check out our notebook for fine-tuning Llama with gradient checkpointing and adapters.

More

• Custom names for AdapterFusion layers (@calpt via Custom names for AdapterFusion layers #774):
  Allows distinguishing multiple fusions on the same adapter by name. See details.
• Allow specifying adapter dtype in AdapterConfig (@killershrimp via Allow specifying adapter dtype in AdapterConfig #767)

Changed

• Major refactoring of library test suite (@TimoImhof via Tests Refactoring #740):
  See tests readme for details.
• Upgrade supported Transformers version (@calpt via Upgrade Transformers to v4.46.x #757; @lenglaender via Upgrade Transformers to v4.47.x #776)

Fixed

• Fix Mistral flash attention with adapters (@divyanshuaggarwal via fix flash attention for mistral.  #758)
• Fix Bottleneck configs to work with ln_before = True and init_weights = "mam_adapter" (@julian-fong via [BUG] Fixes Bottleneck Configs to work with ln_before = True and init_weights = "mam_adapter" #761)
• Fix default Lora/ (IA)^3 scaling in forward (@calpt via Fix default Lora/ (IA)^3 scaling in forward #770) Note: This restores compatibility with adapter-transformers, but changes logic compared to previous Adapters versions. See Fix default Lora/ (IA)^3 scaling in forward #770 for details.
• Fixes for sequence generation & orthogonal projection with ReFT (@calpt via ReFT generate & orthogonal fixes #778)
• Various minor compatibility and warning fixes (@calpt via Readme & test fixes #780, Fix GenerationMixin warning for AdapterModel classes #787)

---

This discussion was created from the release Adapters v1.1.0.