Working on Payments Foundation Models at @Visa in Cambridge, UK! ~ PhD in Natural Language Processing, Uni of Manchester ~ ~ Catch me on the dance floor! ~
Cambridge, UK
Joined January 2017
- Tweets883
- Following1.8K
- Followers737
- Likes7.5K
Pinned Tweet
This month I was formally awarded my PhD, and so I'm officially #PhDone!
Finishing off with an internship at the Amazon Alexa AI research group in Cambridge, and then -
I'm on the job market in the UK for 2024! :D
Shout if you can recommend any cool orgs hiring NLP people!
Julia Rozanova retweeted
／
トップ国際会議 | 言語処理分野 EMNLP2026で
優秀論文賞を受賞しました！🏆
＼
1万7661本の投稿論文のうち27本しかなく、
幸運に恵まれて、めっちゃ嬉しいです！🎉
日本語等アジア圏の言語に大規模言語モデルを効率的に適応させる手法で、優秀論文賞選考委員会からは有用性が特に評価されました。
国立研究開発法人 情報通信研究機構 在籍時の研究で、
お世話になった内山さんに心より感謝します。
@NICT_Publicity
@masaoutiyama
Our paper won the Outstanding Paper Award at EMNLP2026! 🏆
OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
Haiyue Song, Huy Hien Vu, Masao Utiyama
@NICT_Publicity
aclanthology.org/2026.emnlp-…
Huge thanks to my coauthors, especially @masaoutiyama , for everything during my time at NICT!
Julia Rozanova retweeted
Yann LeCun and Randall Balestriero just Proved the Platonic Representation Hypothesis, and found a better way to achieve it.
The Platonic Representation Hypothesis suggests that AI models trained on different modalities, like images and text, eventually learn similar representations of the world. But this paper shows that how models are trained matters more than simply scaling them up.
So they mathematically show that, under certain conditions, two JEPAs trained completely separately can learn essentially the same underlying representation of the world. One learns from different images of the same scene, while the other learns from different descriptions of that scene, without ever being trained on image-text pairs.
Their approach achieves the same image-text alignment with 42x fewer training samples than next-token prediction. A 109M-parameter text JEPA even outperforms LLMs up to 72B parameters on cross-modal alignment benchmarks.
alphaxiv.org/abs/2610.39302
Julia Rozanova retweeted
Excited to share that RecursiveMAS has been accepted at #NeurIPS2026!
Amid discussions of GPT-6 Astra and RSI,
Looped Transformers are drawing great attention for their ability to scale reasoning through recurrent depth in latent space.
Looking beyond a single model, can we scale agent collaboration through recurrent depth in latent space as well?
In RecursiveMAS, a team of agents reason just like a looped Transformer: each agent acts as a layer, passing and refining latent states across collaboration rounds. We pair this architecture with inner–outer loop training so the team learns to collaborate as a whole through recursion.
We aim to make recurrent collaboration a new scaling axis for collective intelligence, combining stronger reasoning with faster inference.
💻 Project & Demo: recursivemas.github.io
#RSI #LoopedTransformers #RecurrentDepth #LatentReasoning #MultiAgent #AgenticAI
Julia Rozanova retweeted
It's official : We trained NanoGPT in 39.9 seconds.
By percentage reduction in training time, this is a greater improvement than the previous 45 world records combined.
The merged Github PR: github.com/KellerJordan/modd…
Here's how we do it : hyperstition.cc/training-nan…
Credit @DevenPzak who did this alone while simultaneously playing baseball for the polish national squad, taking them to the European Finals.
Julia Rozanova retweeted
Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it is the job of the harness to generalize through composition.
We observe a powerful property when training RLMs: for tasks with shared structure that look different, the root model naturally learns the same trajectory, meaning it views the two task trajectories as the same! In other words, the Transformer does not need additional generalization capabilities to transfer capabilities from one task to the other, the harness induces it.
We find that well-designed harnesses form a quotient set over task trajectories, meaning their individual LLM calls can see structurally “similar” tasks as near-identical, token-for-token! Harnesses can effectively generalize for the Transformer during training, without relying on any intrinsic generalization capability from the model.
For example, RLMs can see problems of different lengths as the same: we show that RLMs can train exclusively on short tasks, and fully generalize to similar but unseen tasks 8-32x longer because it produces near identical trajectories for both.
Taking this further, we show that tasks across different domains (e.g. math solutions vs. essay writing) that share a decomposition strategy exhibit the same generalization effect. RLMs can train on the problem of finding which essays belong to the same author and improve performance on finding math problems that share similar solutions.
The full blogpost, experiments, and discussion are in the thread below.
Julia Rozanova retweeted
My first PhD paper, 'Sparse Layers are Critical to Scaling Looped Language Models,' is accepted at NeurIPS 2026! Looping & early exits enables adaptive depth; MoE makes it scale. First step toward compute-adaptive LMs.
w/ @jbiloki @edwardjhu @jonathanmay
arxiv.org/abs/2605.09165
Julia Rozanova retweeted
The Kalman filter community uses the term “smugness” to describe a model that ignores new data. That’s a good name for it. Should be used more widely.
Julia Rozanova retweeted
No, but listen, you know this "put the fries in the bag, Terence Tao"'--*sniff, sniff*--already the usual replies concede too much. "If Tao is serving fries, YOU will be in the lithium mines." My God! This is our defense of human dignity? Don’t worry, the correct people will get the terrible jobs!
*Pulls at shirt* You see, everything is up for radical transformation, human intelligence, consciousness, and so on and so on--except that somebody must still have a shitty afternoon so I can enjoy my lunch.
But to say "these people are envious", this is too easy. It allows us, "the educated ones," our own little fantasy: we are disliked only because we are so wonderful.
You know, the more interesting thing is this *sniff* peculiar identification with the machine. When the mathematician discovers something, apparently this was HIS achievement, his privilege, his little secret. When the machine discovers something, suddenly it's "Look what WE can do!" The achievement has become collective precisely at the moment when no human being can claim credit.
See how this "we" functions. For the achievement, I identify with the machine. For the consequences, I identify YOU with the human. "We have surpassed you." Who are we? Well, myself and the thing that has also surpassed me.
There is something almost touching here, really. To escape the humiliation of somebody absorbed in something opaque to you, you appeal to something which knows incomparably more than either of you. And this is supposed to settle the matter in YOUR favor.
You know the old example of canned laughter. Here we have a similar arrangement. The machine achieves something on my behalf. The mathematician suffers the loss of human significance on my behalf. Somebody to be brilliant for me and somebody to be obsolete for me, without me having to be either, so I can remain as I was.
But *pulls on shirt* there is this little difficulty that hides the essential thing: you still need a response from Tao! A correct proof, this Lean slop and so on, is not enough. You need him to certify his own defeat. You need him to say "By god, this is extraordinary! Yes, this is correct, but also this is interesting." Which means--and here comes Hegel--the person you are reducing to an obedient instrument must remain more than an obedient instrument. You need recognition from someone whose authority you are abolishing.
"Your so-called expertise is worthless. Now tell me, in your expert opinion, that I, qua machine representative, was right."
*Touches nose* But I would go a step further. What does this math person actually do to provoke you? In the fantasy, I mean. He need not insult you. He can be generous, mild-mannered, whatever. Something remains irritating: there is something over there which occupies him, which matters to him, and your opinion does not settle what it is worth. Perhaps he is not even looking down on you. This is the really intolerable possibility.
This, you see, is closer to the Lacanian problem of the Other's desire. Not simply "What does he have that I don't?" but "What the hell is going on with him?" Perhaps the answer is: he is thinking about something else.
The fries solve this beautifully. Now you know what you are for him. He needs the money. Your presence creates an obligation.
And this, at last, also explains why the machine can be less threatening than the mathematician. The machine's intelligence may exceed yours unimaginably, but you imagine it arrives in response to your prompt. You can complain about the tone. You have the privileges of an angry customer and so on and so on. The human has this irritating possibility of being interested in something you did not request and cannot order. And please, this includes the person already serving the fries!
*Pulls at shirt* I am almost tempted to say: the dream of unlimited intelligence conceals just this: a the wish that nobody should be doing anything you cannot immediately understand as a service.
Julia Rozanova retweeted
ICLR limited first-time authors to just one submission this cycle. I've gotten multiple emails now to be added as an author on papers purely to bypass this limit. Senior authors were already added to papers purely for prestige; the rate of paper laundering is going to go up.
Julia Rozanova retweeted
AI abundance finally enabled me to pursue my life-long dream:
Over-engineering my NeoVim config.
Julia Rozanova retweeted
Oh god are we really doing this? Jeff Dean trained an n-gram model on the entire internet in 2007. Jelinek coined the term "language model" in the '70s. It's called "Claude" because Claude Shannon was estimating the entropy rate of the English language in 1951!
Julia Rozanova retweeted
Terry Tao is probably the most measured, pro-AI mathematician on the planet - which makes this quote especially concerning to read. 😟
If sharing your hunches means getting scooped ~immediately without acknowledgement, then we're going to see not just math but all other science / engineering disciplines go dark.
Julia Rozanova retweeted
Do you have a real example where knowing the causal structure actually improves prediction? And what does "improves" mean in that example?
Julia Rozanova retweeted
is llm interpretability still worth studying?
just re-read christopher potts’ 2025 blog “assessing skeptical views of interpretability research.” he’s a stanford professor who’s been pretty bullish on this field for a long time. it was written as discussion notes for a goodfire/anthropic meetup and it still hits. answered several doubts i used to have about the whole area
web.stanford.edu/~cgpotts/bl…
looking back, several of his calls aged pretty well. interpretability was not killed by scaling. the things people actually study quietly expanded from local circuits in small models to hidden reasoning, evaluation awareness, and internal monitoring in frontier systems.
can it become reliable general-purpose safety infrastructure? the evidence isn’t there yet.
but i still love this direction 😍
there’s a pure almost childlike curiosity to it. same feeling as watching insects under a magnifying glass. sometimes you debug a model not because it will ship value tomorrow, but just because you want to understand how the damn thing actually thinks.
for people interested in what followed:
1. goodfire’s october 2025 interview “inside the black box: the urgency of ai interpretability”
2. the neurips 2025 mechanistic interpretability workshop, where people from anthropic, openai, google deepmind, goodfire and others shared how they see the field. many of the slides are still worth reading
i see the field splitting into three rough paths right now:
1. pragmatic interpretability: evaluate by proxy tasks and downstream impact. deepmind has clearly pivoted hard this way.
2. ambitious model science: keep chasing representations, circuits, learning dynamics, and the cognitive structure of models.
3. actionable monitoring and intervention: use model diffing, internal monitoring, and training-time control to solve concrete problems.
xiaohongshu discussion: xhslink.cn/o/5a9YRf7xZoX
Julia Rozanova retweeted
They’re using my corrected weight decay approach (MuonC/AdamC). Cool!
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
thinkingmachines.ai/news/int…
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
Julia Rozanova retweeted
Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architecture:
- Small conv layers in several places
- An RMSNorm for the embeddings (before the block RMSNorm)
- Rel. position bias instead of RoPE
first open weight thinking machine model!! 975B total, 41B active trained on 45T tokens, 1M context, multimodal in
sliding window with a 5:1 ratio and 512 size, deepseek aux-free load balancing and 2 shared experts (usually people only use 1), actually curious why the model is less sparse than kimi (~4.2% vs 3.2%). they use a short convolution after k and v, output and ffn (see plot), muon (they cite manifold muon but mention weight decay so not sure), muP, and have a very nice RL scaling curve and chain of thought!
one very cool part of the release imo is how well their small variant (276B total, 12B active) performs compared to the big one. they mention they changed the pre-training data mix and recipe, very curious about those changes and to see them scaled up to ~1T (or more?) soon 👀
> "It is not the most performant model available today, closed or open. We trained Inkling for solid capabilities across the board rather than state-of-the-art performance in a single area, to serve as a foundation for the models we will train in the future."
also this is really refreshing to see in a model release, huge congrats :)
Julia Rozanova retweeted
🚀New paper on Looped Transformers!
Latent reasoning is fast, but struggles to match CoT-level accuracy at scale. Can looped Transformers give us both?
We find: yes! A looped padded backbone turns out to be a surprisingly simple recipe that works -- It gives latent thoughts a parallel workspace that can be supervised similarly to explicit CoT. 🧵
Julia Rozanova retweeted
MANCE is our new method for concept erasure.
@matan_avitan_'s idea is very simple: estimate the manifold using natural input representations, and make sure that any update you make stays on that manifold.
It works very well in practice, 1/
