Transformer Geometry: Decomposing Representation Updates into Parallel and Perpendicular Subspaces
Why do heavily parameterized attention and MLP sub-layers actively allocate capacity to produce updates aligned with incoming states, when the residual stream already provides identical carryover for free?
Projects directly onto the incoming representation vector \(h\). Geometrically, it modulates the magnitude and scaling of existing semantic features without altering their direction.
Lies strictly orthogonal to \(h\). It drives directional rotation and semantic shifts, steering representations into novel representational subspaces required for complex reasoning.
Updates split into parallel (Δh∥) along hidden state h and perpendicular (Δh⊥) components, scaled via α and β.
Parallel-only scaling is mathematically equivalent to an attention self-aggregation diagonal shift Ai,i ← Ai,i + δ at inference.
Suppressing Δh∥ leaves PPL and reasoning intact (+0.46), whereas perturbing Δh⊥ triggers catastrophic capability loss.
Experience the mathematical duality of Transformer updates in real time: drag the update vector handle directly on canvas, or modulate α (Parallel Scale) and β (Perpendicular Scale) to inspect orthogonal projections, angular deflections, and the resulting perplexity cliff.
Parallel scaling operates with near-zero perplexity impact (+0.00 ΔPPL). Semantic representations stay on the manifold while preserving downstream factual recall.
Systematic empirical and theoretical investigation across models, interventions, head sensitivity, long-context retrieval, compression, and pretraining.
Residual stream updates persistently allocate substantial capacity (12%–25% variance) to direction-preserving components.
Value-space (XSA) parallel suppression is resilient (ΔPPL +0.46), whereas perpendicular perturbation precipitates severe collapse.
Single-head edits cause negligible loss change, whereas joint all-head suppression triggers super-linear degradation.
Full parallel removal preserves downstream reasoning across scales; RULER audits 4k–12k needle retrieval.
| Benchmark | Baseline | No-Para | No-Perp |
|---|---|---|---|
| Core-MC (9 tasks) | 55.28% | 55.19% | 37.24% |
| MMLU (5-shot) | 60.39% | 60.21% | 24.62% |
| GSM8K (Strict) | 45.8% | 45.4% | 8.2% |
| TruthfulQA (MC2) | 48.2% | 48.0% | 31.5% |
| Average | 52.42% | 52.20% | 25.39% |
Pruning disproportionately corrupts Δh⊥, whereas quantization (GPTQ/AWQ) preserves update geometry.
Suppressing parallel attention updates during pretraining consistently yields lower validation loss from 300M to 2.7B.
Interactive evaluation metrics across component scaling sweeps, pretraining convergence, and compression distortion.
Clean Python interfaces for probing geometry, registering dynamic hooks, and running downstream evaluations.
# Clone repository and install dependencies
git clone https://github.com/Shwai-He/transformer-geometry.git
cd transformer-geometry
pip install -r requirements.txt
pip install -e .If you find this research or code useful, please cite our EMNLP 2026 Findings paper:
@misc{he2026disentanglingrepresentationevolutiontransformers,
title={Disentangling Representation Evolution in Transformers through Directional Decomposition},
author={Shwai He and Haichao Zhang and Shen Yan},
year={2026},
eprint={2609.15975},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2609.15975},
}