Christopher Irwin (@christopherIrw) header
Christopher Irwin (@christopherIrw) profile picture

Reasearch Associate at University of Cambridge | Prev. PhD student on AI for life sciences @ UPO Italy

Italia
Joined December 2011
Photo by Christopher Irwin (@christopherIrw)Photo by Christopher Irwin (@christopherIrw)Photo by Christopher Irwin (@christopherIrw)Photo by Christopher Irwin (@christopherIrw)Photo by Christopher Irwin (@christopherIrw)Photo by Christopher Irwin (@christopherIrw)Photo by Christopher Irwin (@christopherIrw)Photo by Christopher Irwin (@christopherIrw)Photo by Christopher Irwin (@christopherIrw)
New preprint out! 🚀 What if sheaf-inspired message passing is a form of multi-head attention where the heads can talk to each other? We explore this connection and what it opens up for graph learning. Read the thread and paper for details! 👇
🚀 New preprint: Let the Heads Talk: Beyond Diagonal Graph Attention arxiv.org/abs/2610.01494 I’m excited to share our latest work, which bridges two worlds that have largely developed in parallel: Sheaf Neural Networks and Attention. 1/4
1
1
188
Christopher Irwin retweeted
In the fullness of time, LLMs will eventually become commodities and their price will be a race to the bottom. As the competitive advantage of one LLM over another shrinks - particularly as open source models expand - the value proposition will turn to the neuro/symbolic harnesses that use those models, structures that encompass agents, agent orchestration, tooling, and component plugins. Even those from frontier AI organizations will fall to open source frameworks. The last thing standing will be those who command the data centers on which these things run, and even then, that will become a commodity, with the number of players shrinking to just a few survivors. They too will find pressure: as computational power rises, more and more such systems will be run locally.
94
181
47
1,413
240,457
Christopher Irwin retweeted
We just got the top community score on ARC-AGI-3's 25 public games: 78.4% and 160/183 levels cleared, where the best frontier LLM session scores 7.8%. No training or demonstrations allowed. Paper: arxiv.org/abs/2607.01531 Blog: david-courtis.github.io/opin… w/ @WenhaoLi29 @ScottSanner #AI #worldmodel #agentic #AGI @arcprize @fchollet
9
17
1
88
13,578
Christopher Irwin retweeted
Remember: Statistics don't lie, but liars use statistics.
OpenAI is ~1-2 generations away from saturating TerminalBench 2.1 When will SWE-bench be saturated... It's getting weird out there folks 🥶
23
21
1
633
154,351
Christopher Irwin retweeted
The AI Scientist: Towards Fully Automated AI Research, Now Published in Nature Nature: nature.com/articles/s41586-0… Blog: sakana.ai/ai-scientist-natur… When we first introduced The AI Scientist, we shared an ambitious vision of an agent powered by foundation models capable of executing the entire machine learning research lifecycle. From inventing ideas and writing code to executing experiments and drafting the manuscript, the system demonstrated that end-to-end automation of the scientific process is possible. Soon after, we shared a historic update: the improved AI Scientist-v2 produced the first fully AI-generated paper to pass a rigorous human peer-review process. Today, we are happy to announce that “The AI Scientist: Towards Fully Automated AI Research,” our paper describing all of this work, along with fresh new insights, has been published in @Nature! This Nature publication consolidates these milestones and details the underlying foundation model orchestration. It also introduces our Automated Reviewer, which matches human review judgments and actually exceeds standard inter-human agreement. Crucially, by using this reviewer to grade papers generated by different foundation models, we discovered a clear scaling law of science. As the underlying foundation models improve, the quality of the generated scientific papers increases correspondingly. This implies that as compute costs decrease and model capabilities continue to exponentially increase, future versions of The AI Scientist will be substantially more capable. Building upon our previous open-source releases (github.com/SakanaAI/AI-Scien…), this open-access Nature publication comprehensively details our system's architecture, outlines several new scaling results, and discusses the promise and challenges of AI-generated science. This substantial milestone is the result of a close and fruitful collaboration between researchers at Sakana AI, the University of British Columbia (UBC) and the Vector Institute, and the University of Oxford. Congrats to the team! @_chris_lu_ @cong_ml @RobertTLange @_yutaroyamada @shengranhu @j_foerst @hardmaru @jeffclune
74
462
98
2,453
975,955
Christopher Irwin retweeted
I'm releasing 53 slides on post-training, covering core algorithms like DPO and GRPO, as well as data quality, synthetic data pipelines, and on-policy training. I had the pleasure of presenting it yesterday as a guest lecturer in Cambridge, UK
16
105
2
1,101
56,016
Christopher Irwin retweeted
How do time series foundation models forecast unseen dynamical systems? In new experiments, we find that small transformers learn to approximate transfer operators in-context. (1/N) arxiv.org/abs/2602.18679
3
76
5
385
30,018
Christopher Irwin retweeted
Very little about software engineering has changed over past last three months. A great deal has changed about coding, not unlike when we saw the rise of high order programming languages and compilers, the difference today being that the number of developers is far larger and distribution channels are such that the velocity and breadth of change is far greater. The entire history of software engineering is one of raising the level of abstraction.
Software engineering changed more in the last 3 months than the preceeding 30 years. Everything about running a software company needs to be rethought from first principles.
91
184
33
1,771
222,451
Christopher Irwin retweeted
Replying to @tensorqt
when the shape rotator tries to prematurely decode into language
1
2
1
48
1,384
New high-effort article "Why Creativity Cannot Be Interpolated" co-written with Dr. Jeremy Michael Budd. Yes the name is a pun on the famous book by @kenneth0stanley! The counterintuitive thesis (corollary of Kenneth's research): - Intelligence and agency are orthogonal to creativity - and sometimes actively hostile to it. - Genuine creativity is impossible without deep understanding and creativity without understanding is "slop". The strangest property of LLMs: within a single frame they seem to comprehend so deeply, yet they possess no perspective of their own. Like the blind men and elephant parable, each report is accurate, yet none integrates. We call this "frame-dependent" understanding, and it will change how you think about AI creativity. We started writing this 2 years ago, and this is our distilled understanding of AI creativity in 2026.
4
19
2
100
7,044
Christopher Irwin retweeted
How do LLMs decide what to predict next? We study this question through the lens of information theory. We show that Entropy-Lens is a simple way to track how the pool of plausible next-token candidates grows and shrinks inside a transformer during prediction.
1
3
8
328
I repeat: We have to split lectures / practice / exams and people should be allowed to pick and mix whatever elements from whatever academic institution. In particular universities must allow LLM-self-taught people to get "real world credentials".
5
5
2
74
6,645
Christopher Irwin retweeted
I've never felt this much behind as a programmer. The profession is being dramatically refactored as the bits contributed by the programmer are increasingly sparse and between. I have a sense that I could be 10X more powerful if I just properly string together what has become available over the last ~year and a failure to claim the boost feels decidedly like skill issue. There's a new programmable layer of abstraction to master (in addition to the usual layers below) involving agents, subagents, their prompts, contexts, memory, modes, permissions, tools, plugins, skills, hooks, MCP, LSP, slash commands, workflows, IDE integrations, and a need to build an all-encompassing mental model for strengths and pitfalls of fundamentally stochastic, fallible, unintelligible and changing entities suddenly intermingled with what used to be good old fashioned engineering. Clearly some powerful alien tool was handed around except it comes with no manual and everyone has to figure out how to hold it and operate it, while the resulting magnitude 9 earthquake is rocking the profession. Roll up your sleeves to not fall behind.
2,554
7,299
2,906
55,182
17,119,922
Hear me out: A question is its answer with noise, a reasoning model is a denoising autoencoder, the reasoning is the embedding Z of the question so that a dumb causal decoder can generate the answer.
26
8
4
179
25,048
Christopher Irwin retweeted
Our massive conferences are increasingly unable to reliably verify findings at scale. This creates a perverse incentive structure that, instead of rewarding truth and accuracy, prioritizes novelty, citation impact, and speed. The body of literature has become like a floating hot air balloon that must collapse in due time.
2
11
1
125
14,128
Christopher Irwin retweeted
it’s not a “bug that leaked the names” when someone obviously spent time taking apart the API to exploit security vulnerabilities and expose names of reviewers. it’s a malicious hack, a deliberate leak, and it is obvious where it came from. now do something about it.
2
3
88
40,519
Christopher Irwin retweeted
A number of people are talking about implications of AI to schools. I spoke about some of my thoughts to a school board earlier, some highlights: 1. You will never be able to detect the use of AI in homework. Full stop. All "detectors" of AI imo don't really work, can be defeated in various ways, and are in principle doomed to fail. You have to assume that any work done outside classroom has used AI. 2. Therefore, the majority of grading has to shift to in-class work (instead of at-home assignments), in settings where teachers can physically monitor students. The students remain motivated to learn how to solve problems without AI because they know they will be evaluated without it in class later. 3. We want students to be able to use AI, it is here to stay and it is extremely powerful, but we also don't want students to be naked in the world without it. Using the calculator as an example of a historically disruptive technology, school teaches you how to do all the basic math & arithmetic so that you can in principle do it by hand, even if calculators are pervasive and greatly speed up work in practical settings. In addition, you understand what it's doing for you, so should it give you a wrong answer (e.g. you mistyped "prompt"), you should be able to notice it, gut check it, verify it in some other way, etc. The verification ability is especially important in the case of AI, which is presently a lot more fallible in a great variety of ways compared to calculators. 4. A lot of the evaluation settings remain at teacher's discretion and involve a creative design space of no tools, cheatsheets, open book, provided AI responses, direct internet/AI access, etc. TLDR the goal is that the students are proficient in the use of AI, but can also exist without it, and imo the only way to get there is to flip classes around and move the majority of testing to in class settings.
Gemini Nano Banana Pro can solve exam questions *in* the exam page image. With doodles, diagrams, all that. ChatGPT thinks these solutions are all correct except Se_2P_2 should be "diselenium diphosphide" and a spelling mistake (should be "thiocyanic acid" not "thoicyanic") :O
917
2,358
484
16,332
2,542,435
I seem to recall a thread with @polynoamial recently arguing that universities should drop much of the "old" AI syllabus and instead just teach modern deep learning methods. But could this be the wrong advice? @pmddomingos argues that, rather, if students spent more time studying Russell and Norvig -- we might save "~100 billion dollars in wasted compute!" 😀🌶️
6
12
2
82
9,718
Christopher Irwin retweeted
Andrej Karpathy calls AI Agents slop "Overall, the models they are not there. And I feel like the industry [...] it's making too big of a jump and it's trying to pretend that this is amazing. And it's not—it's slop! And I think they are not coming to terms with it. And maybe they are trying to fundraise or something like that, I'm not sure what's going on."
266
627
294
7,223
3,275,323
Christopher Irwin retweeted
- you are - a random CS grad with 0 clue how LLMs work - get tired of people gatekeeping with big words and tiny GPUs - decide to go full monk mode - 2 years later i can explain attention mechanisms at parties and ruin them - here’s the forbidden knowledge map - top to bottom, how LLMs actually work - start at the beginning - text → tokens - tokens → embeddings - you are now a floating point number in 4D space - vibe accordingly - positional embeddings: - absolute: “i am position 5” - rotary (RoPE): “i am a sine wave” - alibi: “i scale attention by distance like a hater” - attention is all you need - self-attention: “who am i allowed to pay attention to?” - multihead: “what if i do that 8 times in parallel?” - QKV: query, key, value - sounds like a crypto scam - actually the core of intelligence - transformers: - take your inputs - smash them through attention layers - normalize, activate, repeat - dump the logits - congratulations, you just inferred a token - sampling tricks for the final output: - temperature: how chaotic you want to be - top-k: only sample from the top K options - top-p: sample from the smallest group of tokens whose probabilities sum to p - beam search? never ask about beam search - kv cache = cheat code - saves past keys & values - lets you skip reprocessing old tokens - turns a 90B model from “help me I’m melting” to “real-time genius” - long context hacks: - sliding window: move the attention like a scanner - infini attention: attend sparsely, like a laser sniper - memory layers: store thoughts like a diary with read access - mixture of experts (MoE): - not all weights matter - route tokens to different sub-networks - only activate ~3B params out of 80B - “only the experts reply” energy - grouped query attention (GQA): - fewer keys/values than queries - improves inference speed - “i want to be fast without being dumb” - normalization & activations: - layernorm, RMSnorm - gelu, silu, relu - they all sound like failed Pokémon - but they make the network stable and smooth - training goals: - causal LM: guess the next word - masked LM: guess the missing word - span prediction, fill-in-the-middle, etc - LLMs trained on the art of guessing and got good at it - tuning flavors: - finetuning: new weights - instruction tuning: “please act helpful” - rlhf: reinforcement from vibes and clickbait prompts - dpo: direct preference optimization, basically “do what humans upvote” - scaling laws: - more data, more parameters, more compute - loss goes down predictably - intelligence is now a budget line item - bonus round: - quantization: - post-training quantization (PTQ) - quant-aware training (QAT) - models shrink, inference gets cheaper - gguf, awq, gptq, all just zip files with extra spice - training vs inference stacks: - deepspeed, megatron, fschat, for pain - vllm, tgi, tensorRT-LLM, for speed - everyone has a repo - nobody reads the docs - synthetic data: - generate your own training set - model teaches itself - feedback loop of knowledge and hallucination - welcome to the ouroboros era - final boss secret: - you can learn all of this in ~2 years - no PhD - no 10x compute - just relentless curiosity, good bookmarks, and late nights - the elite don’t want you to know this - but now that you do - choose to act - start now - build the models
16
75
1
864
57,848