Philosophy Behind the Research
Some of my research begins with a technical question and gradually reveals a more personal one. These reflections trace the ideas behind the papers: what their questions have come to mean to me beyond the experiments themselves.
What remains when the visible structure changes?
Pruning changes a model's parameters and can sharply alter its output behavior, yet the hierarchy of its representations reveals that much of its underlying capacity may remain intact. To understand the model, it is therefore not enough to look only at what has been removed; we must ask what has been preserved, and at which level its essence becomes visible.
The same question has accompanied a change in my own research. My standards, interests, and sources of excitement have evolved, and work that once felt sufficient may no longer satisfy me. But beneath that change, the pursuit is remarkably stable: I still value the long process of thinking, the effort to understand things at their roots, and the rare moment when an idea finally becomes clear. The form of enthusiasm changes; its object remains.
What is removed changes the appearance; what is preserved reveals the essence.
What can be dropped without losing what matters?
Attention Drop studies how much of a Transformer's architecture can be removed without erasing the capabilities that matter. Attention and MLP layers are not equally essential, and carefully dropping redundant components can leave the model far more intact than its reduced structure might suggest. Preserving value does not always require preserving everything.
The paper itself taught me a similar lesson. It was rejected repeatedly, and for a long time those decisions left me discouraged. I had allowed external recognition to become entangled with my judgment of the work. Eventually, I learned to loosen that attachment. A review decision is a judgment made at one moment, not the final measure of an idea. Letting go did not mean caring less about the research. It meant no longer insisting that worthwhile work must be recognized immediately. What has genuine value can survive being overlooked and find its moment in time.
Letting go is not giving up on what matters; it is giving up the demand that its value be recognized at once.
What must be released for the whole to move?
Mixture-of-Experts models are designed to use computation sparsely, yet uneven routing can overload a few experts and force every other worker to wait. Capacity-aware inference places explicit bounds on that overload. By dropping or redirecting excess assignments, it relieves the stragglers and allows the system as a whole to move more efficiently without retraining.
Modern life often develops the same imbalance. Commitments accumulate, attention becomes fragmented, and a small number of overloaded responsibilities begin to dictate the rhythm of everything else. The instinct is often to carry them all through greater effort. But sustainable efficiency sometimes requires the opposite: recognizing our limits, redistributing attention, and releasing what no longer deserves to hold the whole system back. Reducing the load is not a retreat from ambition; it is what allows meaningful work and life to keep moving.
Efficiency is not the ability to carry everything, but the judgment to know what must be released.
Does everything deserve the same depth?
Standard Transformers send every token through the same number of layers, even though not every token requires the same amount of computation. Router-Tuning learns where deeper processing is useful while tuning only the routing mechanism. It makes depth a decision rather than a default, preserving computation for the inputs that need it most.
It is easy to confuse seriousness with giving everything our fullest attention. Yet indiscriminate depth can become another form of waste: simple decisions turn into prolonged deliberation, passing concerns become lasting burdens, and reflection slips into rumination. Judgment is not only the ability to think deeply. It is also the ability to recognize which questions deserve that depth and which can be allowed to pass lightly. Selectivity does not diminish thought; it protects our capacity for the questions that truly matter.
Depth is valuable not when applied to everything, but when reserved for what deserves it.
Must every part of us adapt?
Dynamic networks adapt their parameters to each input, but making every parameter dynamic introduces substantial redundancy and deployment cost. PAD-Net separates what truly needs to respond from what can remain fixed. By keeping only essential components dynamic and converting the rest into static parameters, it preserves adaptability without requiring the entire model to change each time its circumstances do.
We are often told that adaptability means continually reshaping ourselves around new environments, expectations, and judgments. But changing in every direction can gradually dissolve the stable sense of self from which meaningful choices are made. A more mature flexibility begins with distinction: principles, commitments, and parts of our identity may deserve continuity, while habits and strategies remain open to revision. We do not adapt well by becoming entirely fluid. We adapt well by knowing where change is necessary and where steadiness gives change its meaning.
Adaptability is not the willingness to change everything; it is the wisdom to know what should remain steady.
What if a mistake reveals another path?
MEO began with a mistake. While developing the work that followed PAD-Net, I wrote the order of summation and multiplication in the expert computation the wrong way around. What first appeared to be an incorrect formula suggested a different execution path: merge the selected experts first, then perform one effective computation. The destination was similar, but changing the order revealed a substantially more efficient way to reach it.
We are trained to recognize mistakes quickly and correct them before they carry us further astray. Most of the time, that instinct is useful. But immediate correction can also close a door before we understand where it leads. Some errors are not merely failures of reasoning; they are departures from habitual reasoning. If we pause before erasing them, they may expose assumptions we did not know we were making. MEO reminded me that discovery does not always begin with being right. Sometimes it begins with taking a wrong step seriously enough to ask why it leads somewhere interesting.
Some mistakes do not take us away from the answer; they reveal another way to reach it.
Can emptiness make room for something larger?
SparseAdapter challenges the assumption that a limited parameter budget requires a uniformly small adapter. Its Large-Sparse design begins with a larger structure, then preserves only a sparse set of connections. Under the same budget, leaving some connections absent can provide more expressive capacity than filling a smaller structure completely.
We often treat empty space as waste. Calendars should be filled, silence should be answered, and every available capacity should be put to immediate use. But a life with no uncommitted space has little room to change shape. Emptiness can hold possibility: time in which an unexpected idea can develop, attention that has not already been claimed, and freedom to grow beyond the dimensions we first imagined. Restraint is not always a reduction of life. Sometimes it is the architecture that allows life to become larger.
What appears empty may be the space that allows something larger to exist.