Understanding and Harnessing Sparsity in Unified Multimodal Models

Core Insights

Three Crucial Discoveries on Multimodal Sparsity

Our systematic probing reveals why traditional uniform compression fails in unified multimodal architectures.

Understanding Asymmetry

Understanding components act as high-level semantic conditioners during generation. They can tolerate up to 50%+ depth/width pruning with negligible impact on visual quality (<1.2% drop on GenEval).

Compression Tolerance VERY HIGH

Generation Fragility

Generation layers directly govern fine-grained continuous visual tokens and diffusion trajectories. Static pruning causes catastrophic collapse (GenEval plunges from 0.652 to 0.518 at 50% pruning).

Static Pruning Tolerance VERY LOW

Sparse MoE Recovery

Neuron activation distributions display distinct sample-dependent sparsity. Converting dense generation MLPs into MoE experts (16→8 or 32→16) recovers full visual fidelity (0.651) while cutting active FLOPs in half!

Quality Retention 99.8% (at 50% FLOPs)

Interactive Simulator

Interactive Multimodal Sparsity Explorer

Adjust the component focus, compression mechanism, and sparsity ratio to visualize real-time parameter changes, benchmark retention, and degradation dynamics.

1. Target Component

2. Compression Strategy

3. Sparsity / Pruning Ratio 50%

0% (Dense) 25% 50% (MoE Optimal) 75% (Extreme)

Component Robustness: High

Understanding components maintain robust semantic representations even when pruned aggressively in generation tasks.

Active Compute 3.5B from 7.0B dense

GenEval Score 0.648 -0.6% relative

MME Score 1885 -2.9% relative

Quality Retention 99.4% near baseline

Multimodal Performance Retention Curve Benchmark Score vs. Sparsity Ratio

[Diagnostic Probe Active]: Pruning 50% of understanding layers via Layer Dropping preserves 99.4% of GenEval fidelity, proving that visual generation requires minimal text-conditioning depth.

Empirical Deep Dive

Comprehensive Probing & Sparsity Analysis

Direct evidence from depth pruning, width reduction, and neuron activation statistics across generation and understanding benchmarks.

Depth Pruning Across Tasks

Fig. 2

Insight: Dropping understanding layers during generation causes virtually zero performance loss, confirming high depth redundancy in text condition encoders.

Dilemma of Compressing Generation

Fig. 3

Insight: Generation components degrade sharply under static pruning, introducing severe visual artifacts, blurriness, and broken object topologies.

Sample-Dependent Neuron Activation

Fig. 4

Insight: Neuron activation ratios show that intermediate MLP layers fire selectively per prompt, establishing the empirical foundation for Mixture-of-Experts (MoE) partitioning.

MoE Sparse Adaptation Quality Recovery

Fig. 5

Insight: BAGEL-MoE (16→8 and 32→16) completely restores photorealistic generation quality, resolving the compression dilemma with 50% active parameters.

Evaluation Results

Comprehensive Multimodal Benchmarks

Quantitative comparison across text-to-image synthesis, visual question answering, and multimodal reasoning benchmarks.

Visual Generation Benchmarks (BAGEL-7B & MoE Variants) GenEval / DPG-Bench / ImageReward

Method	Active Params	GenEval Overall ↑	Single Obj ↑	Two Obj ↑	Color ↑	Position ↑	DPG-Bench ↑
Dense Baseline Full	7.0B (100%)	0.652	0.981	0.742	0.812	0.461	82.4
Layer Drop (Und-50%)	5.2B (74%)	0.648	0.978	0.739	0.809	0.457	81.9
Width Reduction (Und-50%)	5.2B (74%)	0.645	0.976	0.735	0.805	0.453	81.7
Width Reduction (Gen-50%)	5.2B (74%)	0.518	0.892	0.584	0.671	0.312	68.3
BAGEL-MoE (16 → 8)	3.8B (54%)	0.649	0.980	0.740	0.810	0.459	82.1
BAGEL-MoE (32 → 16)	3.8B (54%)	0.651	0.981	0.741	0.811	0.460	82.3

Visual Understanding Benchmarks (MME, MMBench, POPE, TextVQA) Understanding Capacity Retention

Model Variant	Active Params	MME Total ↑	MMBench ↑	POPE (F1) ↑	TextVQA ↑	SEED-Bench ↑
BAGEL Dense Baseline	7.0B	1942.3	78.4	88.6	68.2	72.5
Und-Layer Drop (20% Pruned)	5.9B	1918.5	77.2	88.1	67.4	71.8
Und-Layer Drop (40% Pruned)	4.9B	1856.1	74.8	86.9	65.1	69.4
Und-Width Reduction (20%)	5.9B	1925.7	77.6	88.3	67.8	72.0

Model Hub

Pretrained Sparse MoE Checkpoints

All checkpoints are open-sourced on Hugging Face under the LLM-Drop organization.

16 Experts • Top-8 Active

BAGEL-MoE-7B-GEN-16to8

Partitions intermediate generation MLPs into 16 discrete experts and dynamically routes to top-8 experts per token. Matches full 7B dense generation quality.

Download on Hugging Face

32 Experts • Top-16 Active

BAGEL-MoE-7B-GEN-32to16

Finer-grained expert partitioning with 32 experts and top-16 dynamic routing. Delivers enhanced specialization and peak GenEval performance (0.651).

Download on Hugging Face

Developer Guide

Quickstart & Probing Workflows

Get started with BAGEL-MoE inference or reproduce our depth/width pruning evaluation benchmarks in a few lines of code.

Python 3.10+ / PyTorch 2.1+ / GPU with 40GB+ VRAM

import torch
from inferencer import InterleaveInferencer
from modeling.bagel import Bagel
from modeling.autoencoder import load_ae
from modeling.qwen2 import Qwen2Tokenizer

# 1. Load Pretrained Sparse MoE Checkpoint
model_path = "LLM-Drop/BAGEL-MoE-7B-GEN-16to8"
tokenizer = Qwen2Tokenizer.from_pretrained(model_path)
model = Bagel.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto")
vae_model = load_ae("your_vae_path")

inferencer = InterleaveInferencer(
    model=model,
    vae_model=vae_model,
    tokenizer=tokenizer,
    vae_transform=None,
    vit_transform=None,
    new_token_ids={}
)

# 2. Text-to-Image Generation with 50% Active FLOPs
gen_context = inferencer.init_gen_context()
prompt = "A majestic cyberpunk fox with iridescent turquoise fur standing in a neon-lit rainstorm, cinematic lighting, 8k resolution."
inferencer.update_context_text(prompt, gen_context)
images = inferencer.generate_image(gen_context, cfg_text_scale=4.0)

images[0].save("cyberpunk_fox_moe.png")
print("Image generated successfully with 50% active parameters!")

Bash Shell Script for Layer Dropping Evaluation

# Run GenEval image generation benchmark under 50% understanding layer dropping
bash scripts/eval/bagel/run_geneval_ld.sh   --keep_ratio 0.5   --drop_type block   --skip_mode und

# Run Multimodal Understanding (MME benchmark) with Layer Dropping
bash scripts/eval/bagel/run_vlm_ld.sh   --dataset mme   --keep_ratio 0.8

Neuron Profiling and Width Pruning Execution

# Step 1: Calibrate neuron activation importance scores across understanding tasks
python scripts/neuron_partition.py   --model_path /path/to/bagel   --calibration_task understanding   --calibration_samples 64

# Step 2: Evaluate width reduction on GenEval benchmark
bash scripts/eval/bagel/run_geneval_wr.sh   --keep_ratio 0.5   --target_component understanding

Converting Dense MLP to Sparse MoE Partitions

# Explore interactive walkthrough in Jupyter Notebook:
# notebooks/dense2sparse.ipynb

from utils.moe_utils import convert_dense_to_sparse_moe

# Convert generation MLPs to 16 experts with top-8 sparse routing
sparse_bagel = convert_dense_to_sparse_moe(
    dense_model=model,
    num_experts=16,
    active_experts=8,
    target_modules=["mlp.down_proj", "mlp.gate_up_proj"]
)
print("Dense model converted to Sparse MoE successfully!")

BibTeX

Cite This Work

@misc{he2025understandingharnessingsparsityunified,
  title={Understanding and Harnessing Sparsity in Unified Multimodal Models},
  author={Shwai He and Chaorui Deng and Ang Li and Shen Yan},
  year={2025},
  eprint={2512.02351},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2512.02351},
}