Philosophy Behind the Research
Where Technical Curiosity Meets Personal Reflection
Some research begins with a technical question and gradually reveals a more personal one. These essays trace the ideas behind the papers: what model pruning, dynamic routing, parameter sparsity, representation geometry, and capacity bounds teach us about patience, depth, resilience, and living.
What remains when the visible structure changes?

Pruning removes parameters and can sharply alter a model's output behavior, yet analyzing the hierarchy of its representations reveals that its core geometric capacity often remains intact. To understand a network, it is not enough to measure only what has been stripped away; we must investigate what has been preserved, and at which layer of abstraction its true essence resides.
A parallel shift has unfolded in my own research journey. Over time, my tastes, interests, and sources of excitement have naturally evolved; problems that once felt captivating may no longer hold the same appeal. Yet beneath that visible evolution, the foundational impulse remains remarkably steady: the patience for slow, deliberate thinking, the urge to comprehend problems from their roots, and the quiet joy of the moment an idea finally clicks into place. The outward forms of curiosity change; the underlying pursuit endures.
What is removed alters the appearance; what is preserved reveals the essence.
What can be dropped without losing what matters?

Attention Drop explores how much of a Transformer's architecture can be discarded without eroding the capabilities that define it. Attention and MLP layers do not contribute equally across the network, and selectively dropping redundant components leaves the model's core intelligence surprisingly intact. Preserving essence does not require preserving everything.
The paper itself became an unexpected teacher of that lesson. It faced repeated rejections, and for a long stretch, each setback cast a shadow of discouragement over the work. In hindsight, I had allowed external peer validation to become entangled with my intrinsic conviction. Letting go of that anxiety did not mean caring less about the research; it meant relinquishing the demand for immediate approval. True value does not vanish when overlooked—it endures quietly until its moment arrives.
Letting go is not giving up on what matters; it is giving up the demand that its value be recognized at once.
What must be released for the whole to move?

Mixture-of-Experts architectures achieve efficiency through conditional computation, yet unbalanced routing frequently overloads a handful of experts, forcing the entire distributed fleet to stall behind stragglers. Capacity-Aware Inference enforces explicit bounds on expert workload—dropping or reallocating excess tokens so that the collective system regains fluent throughput without requiring expensive retraining.
Life often falls into the exact same imbalance. Commitments quietly compound, attention becomes splintered, and a few overloaded obligations begin to bottleneck our entire momentum. The instinctive reaction is to double down and carry everything through sheer endurance. Yet sustainable efficiency demands the opposite wisdom: recognizing structural limits, redistributing focus, and letting go of what is holding the broader whole back. Setting down excess weight is not a surrender of ambition; it is what allows meaningful pursuits to keep moving forward.
Efficiency is not the capacity to carry everything, but the discernment to know what must be released.
Does everything deserve the same depth?

Standard Transformers push every input token through the uniform depth of all layers, regardless of how simple or complex the token may be. Router-Tuning transforms depth from a rigid default into an intentional decision by training only lightweight routing mechanisms. It dynamically allocates computational depth to difficult tokens while allowing straightforward ones to pass through swiftly.
It is easy to mistake thoroughness for giving every single detail our deepest energy. Yet uncalibrated depth quickly degenerates into exhaustion: trivial choices morph into agonizing deliberations, fleeting concerns become enduring anxieties, and reflection slips into unproductive rumination. Intellectual maturity is not merely the power to think deeply, but the discernment to know which questions warrant that depth and which should be allowed to pass lightly. Selectivity does not diminish thought; it protects the mental clarity required for what truly matters.
Depth is valuable not when forced upon everything, but when reserved for what deserves it.
Must every part of us adapt?

Dynamic neural networks adapt their parameters to each input, yet making every weight dynamic incurs immense parameter redundancy and operational overhead. PAD-Net decouples what genuinely needs to respond from what can remain constant. By preserving dynamism only in crucial sub-modules while fixing the rest, it retains adaptive agility without destabilizing the network's foundational parameters.
We are continually urged to remain endlessly adaptable—reshaping our attitudes and priorities to conform to every shifting trend and external expectation. Yet adapting in every conceivable direction gradually dissolves the coherent core from which meaningful choices originate. True resilience begins with distinction: our foundational principles, core values, and intellectual integrity must remain grounded, while our tactical strategies and habits stay flexible. We do not navigate the world well by becoming entirely fluid; we navigate it by understanding where change is necessary and where steadfastness gives that change purpose.
Adaptability is not the willingness to change everything; it is the wisdom to know what should remain steady.
What if a mistake reveals another path?

MEO originated from a serendipitous mistake. While building the theoretical formulation following PAD-Net, I accidentally reversed the algebraic order of summation and multiplication in the expert routing computation. What initially looked like an erroneous equation unveiled an entirely new operational paradigm: merge the selected experts into a unified weight matrix first, then execute a single matrix multiplication. The computational destination remained equivalent, but the inverted sequence unlocked an order-of-magnitude leap in execution efficiency.
We are trained to treat mistakes as failures to be swiftly erased and forgotten. Most of the time, that discipline keeps us on track. Yet premature correction can close an unexpected doorway before we understand where it might lead. Certain errors are not mere lapses in logic; they are spontaneous departures from habitual orthodoxy. If we pause before discarding them, they can illuminate subconscious assumptions we were never aware of making. Scientific discovery does not always begin with being right from the outset; sometimes it begins with taking an anomaly seriously enough to ask why it leads somewhere intriguing.
Some mistakes do not lead us away from the truth; they reveal an unexpected doorway into it.
Can emptiness make room for something larger?

SparseAdapter challenges the conventional assumption that parameter efficiency demands uniformly downscaled, dense bottlenecks. Its Large-Sparse architecture projects representations into a much wider dimensional space while constraining connectivity through extreme sparsity. Under an identical parameter budget, intentional voids between neurons provide significantly richer representational expressiveness than densely packing a constricted space.
In modern culture, empty space is often perceived as wasted opportunity. Calendars must be packed, silence must be filled, and every reserve of energy is expected to be expended immediately. Yet a life devoid of uncommitted space loses its capacity to accommodate the unexpected. Emptiness is not absence—it is latent potential: the unstructured hours in which spontaneous insights ignite, the unhurried attention that listens deeply, and the psychological margin to pivot toward unforeseen possibilities. Restraint is not a diminishment of life; it is the architectural space that allows something greater to unfold.
What appears empty is often the very space that allows something larger to exist.
Does seeing and reading require a divided mind?

Unified multimodal models aim to process diverse sensory streams—vision, audio, and text—within a single monolithic network, yet forcing varied modalities through identical dense parameters causes severe representational interference. By investigating and harnessing modality-specific parameter sparsity, we find that different senses can naturally share a unified model: activating distinct, specialized neural circuits while drawing upon a shared cognitive foundation.
We often feel pressured to compartmentalize our lives—treating our creative pursuits, analytical routines, and emotional convictions as if they belonged to entirely separate personas. Yet genuine selfhood does not require fractured division. Just as a unified multimodal network comfortably embraces vision and language through sparse, harmonic pathways, our multifaceted interests can thrive within a single coherent identity, lighting up naturally in response to whatever world we encounter.
A unified mind does not force everything into conformity; it leaves room for different faculties to resonate without collision.
Which direction are you actually moving in high-dimensional space?

As representations evolve across deep Transformer layers, standard metrics rely on raw Euclidean distance, conflating mere norm inflation with genuine conceptual progression. Directional decomposition disentangles vector magnitude from angular trajectory, revealing that representation evolution is governed by a small number of persistent semantic axes rather than erratic high-dimensional turbulence.
In modern culture, frantic motion is frequently mistaken for meaningful progress. We exhaust ourselves maintaining momentum without pausing to examine the orientation of our trajectory. Directional decomposition offers a vital anchor: the sheer magnitude of your effort matters little if the underlying angle is drifting. True achievement is not about moving faster; it is about having the discernment to know which principal direction you are truly steering toward.
Magnitude measures the intensity of motion; direction reveals whether you are truly arriving.
How much can you compress before losing individuality?

Mixture-of-Experts architectures attain massive parameter capacity by training specialized sub-networks, but their distributed memory footprint creates formidable deployment barriers. By conducting a holistic study across pruning, quantization, expert merging, and knowledge distillation, we mapped the precise trade-off frontier between preserving rare expert specialization and achieving ultra-compact runtime efficiency.
Over time, we collect countless experiences, roles, and memories. The temptation is to hoard every fragment, fearing that letting go of any piece diminishes who we are. Yet true intellectual maturity is an act of distillation: discerning which core principles define our unique character and condensing them into a compact, durable essence that we can carry lightly into whatever comes next.
Maturity is not the accumulation of every detail, but the distillation of what is essential into something you can carry.
Can a generative mind also be a sharp finder?

Large language models excel at autoregressive generation, yet their deep decoder representations are traditionally deemed too heavyweight for ultra-fast dense retrieval. EffiR bridges this divide by distilling deep generative representations into compact retrieval embeddings, proving that the generative reasoning power of foundation models can be harnessed for high-throughput semantic search without sacrificing retrieval fidelity.
We live in an era overwhelmed by information, where creating new noise is easy but locating deep, timeless truth is increasingly rare. A brilliant mind is not merely one that generates endless commentary; it is one that possesses the discipline to search deeply, filter through the deluge, and retrieve what truly matters. Synthesis and retrieval are twin virtues—before we can create with wisdom, we must first learn how to find with precision.
To create with clarity, one must first master the art of finding what is essential.
How can perception stay nimble without carrying excess weight?

Dynamic convolutions dynamically synthesize custom kernel weights for every visual input, but naive implementations incur severe parameter overhead. SD-Conv demonstrates that dynamic adaptability can be achieved with extreme parameter frugality—adjusting perceptual receptive filters on the fly through lightweight decomposition rather than expanding parameter capacity.
When confronting an unpredictable world, our instinct is to prepare for every contingency by hoarding resources, planning every scenario, and building rigid defensive structures. Yet true adaptability is not about over-preparing; it is about remaining agile. When our core mindset is unburdened by excess baggage, we can adjust our perspective instantaneously to changing light and unfamiliar terrain.
Agility is not having an answer for everything in advance; it is being light enough to adjust in the moment.
