Findings of EMNLP 2022 Parameter-Efficient Fine-Tuning (PEFT) 0.1% ~ 0.5% Parameters

SparseAdapter:
An Easy Approach for Improving the Parameter-Efficiency of Adapters

Shwai He¹ • Liang Ding² • Daize Dong³ • Miao Zhang^4,5 • Dacheng Tao^3,2

¹University of Maryland, College Park ²JD Explore Academy ³The University of Sydney ⁴King's College London ⁵Harbin Institute of Technology

ACL Anthology arXiv:2210.04284 Code & Checkpoints Sparsity Calculator

Executive Summary

While standard adapter tuning maintains efficiency by freezing pre-trained language models, scaling adapter capacity traditionally incurs an undesirable parameter footprint. SparseAdapter proves that dense adapters suffer from significant intra-module parameter redundancy. By integrating dynamic and magnitude-based pruning directly into the adapter bottleneck, SparseAdapter achieves competitive or superior performance with 50% to 80% pruned adapter weights (only 0.1%–0.5% total model parameters).

Why SparseAdapter?

Reimagining parameter-efficient tuning through structured parameter sensitivity and sparsity.

Parameter Redundancy

Systematically reveals that standard dense adapter weights contain heavy redundancy and can be pruned substantially without performance degradation.

Multi-Pruner Support

Natively accommodates Magnitude, SNIP, GraSP, SynFlow, and Random pruning schemes under unified training loops.

Large-Sparse Paradigm

Scales bottleneck dimension $m$ (e.g. 128) under high sparsity (75%), providing higher expressive power under identical parameter budgets.

Beats Full Fine-Tuning

SparseAdapter-L surpasses full fine-tuning on the GLUE benchmark (86.58 vs 86.35) while tuning only 0.36% of the model parameters.

Interactive Tool

Interactive Parameter & Sparsity Calculator

Configure your backbone model, adapter architecture, bottleneck, and sparsity ratio to evaluate exact parameters and estimated GLUE score.

1. Base PLM Backbone 125M params

2. Adapter Architecture Attn + FFN (Houlsby)

3. Bottleneck Dimension ($m$) m = 64

4. Sparsity Ratio ($s$) 50% Pruned

0% (Dense) 25% 50% 75% (Sparse-L) 90%

Active Trainable Params

442.4 K

0.354% of backbone

Savings vs. Full FT

99.65%

-50.0% vs Dense Adapter

Projected GLUE Performance

Optimal Efficiency

86.08

GLUE Average +0.58 vs Dense

MNLI

87.5

QQP

91.8

QNLI

92.7

SST-2

94.6

Standard Sparsity Regime: 50% pruning on $m=64$ preserves full representation capacity while slashing trainable parameters in half, beating standard dense Pfeiffer adapters.

GLUE Benchmark Evaluation

Comprehensive evaluation across 8 NLU tasks using RoBERTa-base (125M backbone).

GLUE Score vs. Param %

RoBERTa-base

Note how SparseAdapter-L ($m=128$, 75% sparsity) establishes a new Pareto-optimal frontier, surpassing full fine-tuning with only 0.36% parameters.

Task Performance Breakdown

Higher is better (↑)

Method	Params	MNLI	QQP	QNLI	SST-2	CoLA	MRPC	Avg
Full Fine-Tuning	100%	87.6	91.9	92.8	94.8	63.6	90.2	86.35
Houlsby (Dense $m=64$)	0.71%	87.2	91.5	92.6	94.2	62.4	89.5	85.50
Pfeiffer (Dense $m=64$)	0.36%	87.0	91.4	92.4	94.0	61.8	89.0	85.06
SparseAdapter (Mag 50%)	0.36%	87.4	91.7	92.7	94.5	63.1	90.0	85.96
SparseAdapter (SNIP 50%)	0.36%	87.5	91.8	92.7	94.6	63.3	90.1	86.08
SparseAdapter-L (75% Sparse)	0.36%	87.8	91.9	93.0	94.9	63.8	90.7	86.58

Methodology & Architecture

Integrating connection sensitivity and dynamic weight pruning within adapter bottleneck layers.

Architecture Diagram View High-Res

Figure 1: Overview of SparseAdapter. Pre-trained transformer parameters remain completely frozen. The inserted adapter bottleneck layers undergo gradient/magnitude-based pruning, removing redundant connections while expanding representational capacity under constrained parameter budgets.

1 Bottleneck Formulation

Adapter modules project the hidden state down to bottleneck dimension $m$, apply non-linearity $\sigma$, and project back to dimension $d$:

$$\mathbf{h}' = \mathbf{h} + \sigma(\mathbf{h} \mathbf{W}_{\text{down}}) \mathbf{W}_{\text{up}}$$

2 Sensitivity Masking

A binary mask $\mathbf{M}$ is computed using connection sensitivity scores $S(w)$ evaluated on task batches:

$$\mathbf{M}_{ij} = \mathbb{I}\left(S(w_{ij}) \ge \tau_s\right)$$

3 Large-Sparse (L)

Increasing bottleneck width ($m \to 2m$) while applying higher sparsity ($s=75\%$) yields equal parameter size with richer subspace expressivity:

$$\text{Params} = 2 d m (1-s)$$

Quickstart & Reproducibility

Run SparseAdapter across tasks with straightforward shell scripts and commands.

# Run SparseAdapter on GLUE MRPC with SNIP pruner and 50% sparsity
python examples/pytorch/text-classification/run_glue_sparse.py \
  --model_name_or_path roberta-base \
  --task_name mrpc \
  --do_train --do_eval \
  --per_device_train_batch_size 48 \
  --per_device_eval_batch_size 48 \
  --learning_rate 1e-4 \
  --num_train_epochs 10 \
  --attn_mode adapter \
  --attn_bn 64 \
  --ffn_mode adapter \
  --ffn_bn 64 \
  --pruner snip \
  --sparsity 0.5 \
  --output_dir ./checkpoints/roberta-base/mrpc/snip_0.5

# Run SparseAdapter on SQuAD Question Answering
python examples/pytorch/question-answering/run_qa_sparse.py \
  --model_name_or_path roberta-base \
  --dataset_name squad \
  --do_train --do_eval \
  --per_device_train_batch_size 32 \
  --learning_rate 3e-4 \
  --num_train_epochs 8 \
  --attn_mode adapter \
  --attn_bn 64 \
  --ffn_mode adapter \
  --ffn_bn 64 \
  --pruner mag \
  --sparsity 0.5 \
  --output_dir ./checkpoints/roberta-base/squad/mag_0.5

# Run SparseAdapter on XSum Abstractive Summarization
python examples/pytorch/summarization/run_summarization_sparse.py \
  --model_name_or_path facebook/bart-large \
  --dataset_name xsum \
  --do_train --do_eval \
  --per_device_train_batch_size 16 \
  --learning_rate 1e-4 \
  --attn_mode adapter \
  --ffn_mode adapter \
  --pruner snip \
  --sparsity 0.5 \
  --output_dir ./checkpoints/bart-large/xsum/snip_0.5

# Environment Setup
conda create -n sparseadapter python=3.8 -y
conda activate sparseadapter

git clone https://github.com/shwai-he/SparseAdapter.git
cd SparseAdapter

pip install -r requirements.txt

Citation

@inproceedings{he2022sparseadapter,
  title     = {SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters},
  author    = {He, Shwai and Ding, Liang and Dong, Daize and Zhang, Miao and Tao, Dacheng},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  month     = dec,
  year      = {2022},
  address   = {Abu Dhabi, United Arab Emirates},
  publisher = {Association for Computational Linguistics},
  pages     = {2184--2190},
  url       = {https://aclanthology.org/2022.findings-emnlp.160/},
  doi       = {10.18653/v1/2022.findings-emnlp.160}
}