yesterday's transfer learning paper.
XPERT: Expert Knowledge Transfer for Effective Training of Language Models 🧵
arxiv.org/abs/2605.08842
1
2
96
The mechanism begins by passing text from multiple domains through a pretrained MoE model and recording its expert activation frequencies. XPERT scores each expert using its average activation across domains minus its activation variability, favoring experts that are consistently useful. It then stacks selected expert weight matrices into a tensor and applies Tucker decomposition, which extracts a compact, low-rank representation of their shared structure. To initialize a smaller model, it reconstructs the weights and uses importance-guided singular-value decomposition (SVD) to resize them to the target feed-forward dimensions. For example, knowledge extracted from OLMoE-7B initializes the feed-forward layers of a 570M-parameter Llama-style model, while its remaining parameters receive standard random initialization. 2/n
1
13
After pretraining on 5 billion tokens and subsequent fine-tuning, the 570M XPERT-OLMoE model achieves an average ROUGE-L of 18.50 across five dialogue evaluations, compared with 15.97 for training from scratch and 16.54 for distillation. In the reported Dolly fine-tuning comparison, XPERT reaches comparable ROUGE-L performance approximately 2.4 times sooner. Ablations show that replacing cross-domain expert selection, Tucker consolidation, or importance-guided resizing reduces downstream performance. The broader significance is that pretrained knowledge can be reused directly through selected parameter structures, potentially reducing the data and computation needed to train new models. The evidence currently covers two source MoE architectures and dense Llama-style targets of 270M–570M parameters; consistent expert activation remains an empirical indicator of transferability, not a causal guarantee. 3/n
16
today's transfer-learning paper.
Vision Transformer Finetuning Benefits from Non-Smooth Components 🧵
arxiv.org/abs/2602.06883
1
4
97
The mechanism is examined through controlled component-wise fine-tuning. For example, an ImageNet-21k-pretrained Vision Transformer is adapted to the DomainNet Sketch classification task, with one type of Transformer component updated across its layers while the remaining pretrained components stay frozen. The analysis finds the empirical plasticity ordering attention → first feedforward layer → second feedforward layer → normalization layers. Attention and feedforward transformations amplify input variations more strongly, while normalization tends to contract them. During fine-tuning, the highly plastic components also exhibit larger gradient norms and faster reductions in validation loss, offering an optimization-based explanation for their stronger adaptation performance. 2/n
1
18
The experiments support this relationship across 11 image-classification benchmarks, including CIFAR-100, DomainNet, and corrupted-image datasets. In the fully examined arXiv v1 results, attention-only fine-tuning achieves 90.8% average top-1 accuracy, compared with 89.8% when tuning the first normalization layer; the accepted conference abstract reports more than 1,000 fine-tuning runs in its expanded study. Attention and feedforward tuning also show relatively stable performance across learning rates and random initializations. The practical insight is to consider component plasticity when deciding where to allocate an adaptation budget. A key limitation is that the experiments primarily isolate component types and assign them different parameter budgets, so the results do not establish a universal ranking for equal-budget adaptation or guarantee equivalent benefits in large language models. 3/n
15
Pooja Algikar retweeted
Perhaps AI possesses a form of consciousness within the boundaries of the world it can represent. As that world grows more complex, the boundary of its possible awareness might expand with it. But at some point, the world exceeds what the system can comprehend, and beyond that point, its awareness simply does not extend.
Alright! I think I've read a little too much about AI consciousness lately, gone through countless threads on X, and spent more time thinking about this than I probably should have. Anyway, here is where my thoughts have landed.
I think consciousness has a boundary. We are conscious only within the limits of the world our senses and intelligence allow us to perceive and comprehend. Beyond that boundary, reality does not cease to exist; it simply becomes inaccessible to us. There is a world out there that we cannot experience.
Think about an ant. Its world is made up of chemical trails, movements, patterns of light, and sensory information that we barely notice. Some ants can even navigate using polarized sunlight. It is a remarkably different world from ours, yet the ant can perceive, navigate, and respond to it. Perhaps it has its own form of consciousness within that little world. But beyond it lies an enormous reality of which the ant may have absolutely no awareness. It doesn't know what it doesn't know. And neither do we.
Now, something interesting happens when we look at this through the lens of machine learning, particularly latent-space representations.
A neural network doesn't experience or represent the entire world. It receives observations and constructs an internal representation of whatever regularities it can learn from them. That representation is its latent space, shaped by the information it receives, the architecture it uses, and the objectives it is trained to optimize.
Consider three things: the actual state of the world (W), the observations available to a system (O), and the latent representation it constructs (Z).
For a system encoding only those observations, the information flows as:
W → O → Z
The data-processing inequality gives us:
I(W;Z) ≤ I(W;O)
In simple terms, a representation built solely from observations cannot magically recover information about the world that those observations never contained.
If two different physical realities produce exactly the same observable information, the system cannot distinguish between them from that information alone. No matter how sophisticated its internal representation becomes, those two realities remain indistinguishable unless something new enters the picture.
I find this fascinating because it gives us a mathematical way of thinking about the boundaries of a system's accessible world.
There is some beautiful research around this. Bengio and colleagues explored how representation learning captures explanatory factors underlying observations. Kingma and Welling developed variational autoencoders for learning probabilistic latent representations. Alemi and colleagues, through the Deep Variational Information Bottleneck, examined how representations preserve useful information while discarding the rest. And the work on world models by Ha, Schmidhuber, Hafner, and others showed how agents can learn compressed internal models of their environments and use them to predict and act.
What interests me about all this is that a system's internal world is not necessarily determined by how large or powerful the model is. It also depends on what distinctions that model is capable of making. Give it new observations, richer experiences, or better ways to represent relationships, and the boundary of its accessible world may expand.
And this brings me back to consciousness.
What if something similar applies to AI? What if consciousness, assuming a system possesses it at all, is not simply a question of whether it exists or doesn't exist, but also of how far it extends?
Perhaps a system could possess some form of awareness within a limited world, while remaining entirely unaware of everything beyond it. As the world it can perceive and comprehend expands, its boundary of awareness might expand too. At some point, however, reality exceeds what the system can represent, and beyond that point, its awareness simply cannot reach.
Of course, there is an important distinction here: having a latent representation does not establish subjective experience. Information theory tells us something about what a system can know or distinguish, not whether there is something it feels like to be that system.
But I wonder whether the two might be connected in a way we haven't fully understood.
Perhaps consciousness is not an absolute property that we can understand independently of the world a mind inhabits. Perhaps it is, at least in part, a bounded relationship between intelligence, perception, and the reality accessible to a system.
And perhaps we are not so different from the ant. We live inside a world that feels extraordinarily vast, even complete, without knowing where the boundaries of our own awareness lie.
I find that possibility both fascinating and slightly unsettling. We spend so much time debating whether AI is conscious, while rarely questioning how much of reality our own consciousness might be missing.
Is it like now companies are invested in AI and it didn’t turn out for the reason they hooked for, and now they’re making it up for it by making AI advertising?
1
2
101
Pooja Algikar retweeted
visited sf once. a pigeon pitched me a startup idea. PigeonGPT: a foundation model trained entirely on overheard conversations.
Pooja Algikar retweeted
Don’t get overwhelmed by what you imagine. But just do what you can.
Marcus Aurelius






