Neil Thomas (@countablyfinite) header
Neil Thomas (@countablyfinite) profile picture

Building AI for biological design @biohub (fka @evoscaleai) 🧬 Prev: @Theteamatx 🌘 PhD @UCBerkeley šŸ™ˆ

Act V, Scene II
Joined November 2015
Photo by Neil Thomas (@countablyfinite)Photo by Neil Thomas (@countablyfinite)Photo by Neil Thomas (@countablyfinite)Photo by Neil Thomas (@countablyfinite)Photo by Neil Thomas (@countablyfinite)Photo by Neil Thomas (@countablyfinite)Photo by Neil Thomas (@countablyfinite)Photo by Neil Thomas (@countablyfinite)Photo by Neil Thomas (@countablyfinite)
Neil Thomas retweeted
Please help spread the word! We are recruiting multiple Postdoctoral Fellows as part of the recently launched Bakar Computational Biomedicine Initiative (BCBI) at UC Berkeley and UCSF. BCBI website: bcbi.berkeley.edu/ Apply by Nov 1, 2026: berkeley.infoready4.com/#fre… (1/n)
4
74
171
23,648
We built a world model of protein biology. @awfderry & I are hosting a webinar on some of our team's interpretability work. We'll walk through the @biohub ESM Atlas live and show how to run the tools on your own proteins. Tue Jul 21, 1pm PT bit.ly/3SWpWAU biohub.ai/esm/protein/atlas
Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology. The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics. We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity. We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures. ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences. A world model of protein biology emerges through language modeling. We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins. The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science. This understanding emerges without prior knowledge, just from language modeling of protein sequences. Language models are becoming a powerful substrate to understand and program biology. The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders. I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.
1
5
12
3,225
Neil Thomas retweeted
I am thrilled to share that UC Berkeley and UCSF have launched a joint initiative in Computational Biomedicine! cdss.berkeley.edu/news/uc-be… We will soon be recruiting new faculty and postdoctoral fellows. Please repost to help spread the word.
2
56
2
239
37,502
Neil Thomas retweeted
Introducing A2D2 šŸƒšŸ”®, our new framework for principled reward alignment of variable-length discrete diffusion models that parameterize insertions of tokens at arbitrary positions along the sequence. šŸ“ƒĀ Paper: arxiv.org/abs/2606.13565 šŸ”®Ā Project Page: sophtang.github.io/adaptive-… šŸ’»Ā Code: github.com/sophtang/A2D2 Some highlights: šŸƒ We derive the Radon–Nikodym derivative for the joint insertion–unmasking path measures, enabling theoretically guaranteed convergence to the reward-tilted sequence distribution. šŸƒ We establish unmasking and insertion quality as tractable approaches for minimizing decoding error, and train lightweight quality predictors alongside the policy. šŸƒ We introduce the Adaptive Joint Decoding (AJD) loss, which provably yields the optimal path measure that generates the reward-tilted distribution while remasking low-quality tokens and dropping low-quality insertions at inference. šŸƒ Empirically, A2D2 improves reward optimization while enhancing generation flexibility and accuracy over prior fixed-length fine-tuning and inference-time guidance methods. More in the thread šŸ‘‡šŸ»
2
8
1
73
8,060
Neil Thomas retweeted
Antibody LMs learn what looks antibody-like, but not how selection turns naive germline antibodies into strong binders. @aakarshv1 and I are excited to share CoSiNE, a model that learns this germline-to-mature process for variant effect prediction and antibody design. (1/8)
8
50
7
228
48,031
Screen 1M random protein sequences to discover that biology-like folds are accessible from random sequences with surprising frequency @KlaraH_lab
3
56
4
315
29,627
Neil Thomas retweeted
🧵 around the interpretability work that helps connect ESMC embeddings to natural language - protein function at the micro level is around residue level mutations but at the macro level is around how they behave in the real world.
One early finding: evolutionary links between gene-editing enzymes across completely different branches of life — connections nobody had made before. This is what becomes possible when you can question protein space at scale, not just search it. Explore ESM Atlas: bit.ly/4dJcF6G
3
16
69
7,939
Neil Thomas retweeted
How to design your own PD-1 binder in 4 easy steps: 1. Download the tutorial notebook from the ESM team 2. Get a @modal API key to scale it up 3. Scaling it up, O($1000) will get you a 96 well plate of minibinders with >50% success rates on typical targets 4. Test it in the lab!
5
15
3
91
12,032
Neil Thomas retweeted
Characterizing AI-designed proteins requires quantitative biochemistry at massive scale. Enter Amplicon/Protein Bead Display (APB-Display), a fully in vitro platform that quantifies Kd's for >100,000 variants in <3 days (preprint link below!) @Stanford_ChEMH @czbiohub (1/n)
3
95
8
440
64,582
Neil Thomas retweeted
I’m so excited about the launch of ESMFold2, ESMC, and the new ESM Atlas. This was a massive team effort, and I’m grateful to have worked with such an incredible group @biohub. A headline result I’m especially excited about: ESMFold2 can design minibinders and antibodies with nanomolar affinity, target selectivity, and functional activity against therapeutically relevant targets. Today, we’re sharing the full binder design protocol.
Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology. The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics. We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity. We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures. ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences. A world model of protein biology emerges through language modeling. We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins. The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science. This understanding emerges without prior knowledge, just from language modeling of protein sequences. Language models are becoming a powerful substrate to understand and program biology. The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders. I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.
3
24
5
84
149,659
A few edible plants have proteins that sit close to miraculin in the ESM Protein Atlas, so I thought I'd try extracting what protein I could from said plants and tasting it... Anyway, null result but an excuse to muck about :) Video lab notes: youtube.com/watch?v=mwGZb8zw…
5
10
3
77
11,826
Neil Thomas retweeted
I added ESMFold 2 to github.com/hgbrian/foldism -- and some other niceties like an optional reference pdb
1
15
82
3,496
Neil Thomas retweeted
We built a joint experimental and computational platform for scalable multi-modal single-cell chemical screens — profiling RNA, protein (including phospho-signaling), and chromatin accessibility responses to thousands of small molecule perturbations in parallel. biorxiv.org/content/10.64898…
2
39
1
179
14,135
Neil Thomas retweeted
We've done a million of these deep dives into interpreting and understanding ESMC features, it's just that we don't quite know how to write about them other than to say "here are a bunch of cool observations".
Here are all the features that activate on the second arginine in the motif - clearly calling out a relationship to phosphoinositide.
1
2
2
29
6,101
Neil Thomas retweeted
Finally got the BDBV Trimer folded properly using esmfold2!
5
34
1,589
It looks like that made a big difference. It found a much higher confidence pose (0.85 ipTM vs 0.81 with AF3 and 0.8 earlier with ESM2) that actually makes much more sense than the original pose and plausibly explains its MOA. Also no artifacts. Amazing work!
3
14
1,756
Neil Thomas retweeted
One feature of the @biohub ESM C release that I think deserves more attention is the interpretability of its latent space. There has been a lot of discussion about whether interpretability is useful for scientific ML models. I think it can become very useful, especially when AI agents can use a model’s internal representations to reason about biology. Here is one example of an AI agent with access to ESM C SAE features correctly interprets the loss-of-function mechanism behind a variant. There is still a lot to improve in how AI agents use model interpretability, but this is an exciting direction for AI agents that don’t just make predictions, but inspect learned representations to generate mechanistic hypotheses. Read more in our blog: aiscientist.tools/posts/ai-a… We've also released the SAE-enabled skills for variant interpretation, loss-of-function analysis, structural annotation, functional mechanism interpretation, and evaluation against experimental datasets via ToolUniverse @ScientistTools Thanks to the team behind this! @GaoShanghua @_yepeng @marinkazitnik @countablyfinite @HarvardDBMI @harvardmed @Harvard @KempnerInst
2
20
1
203
12,953
Neil Thomas retweeted
Super excited to share what we've been working on! ESMC/ESMFold2 show that protein language modeling learns the principles of protein biology and can be used for state-of-the-art structure prediction and design. We also built an interactive atlas of over 6.8 billion proteins!
Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology. The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics. We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity. We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures. ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences. A world model of protein biology emerges through language modeling. We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins. The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science. This understanding emerges without prior knowledge, just from language modeling of protein sequences. Language models are becoming a powerful substrate to understand and program biology. The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders. I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.
1
7
1
34
2,547
Neil Thomas retweeted
I think one of the coolest results from our paper is how much biological information falls out of a masked language modeling objective. Protein-protein interactions, contact maps, and enzyme function can all be extracted from the ESMC’s internal representations. Even more interesting is that these patterns are not obvious from sequence alone. Functionally similar proteins with very different sequences will activate the same SAE features. Endonucleases from opposite ends of the tree of life cluster together in latent space. A single feature activates on the primary catalytic motif across radically diverse proteases. Why is this? Protein language models are, at their core, powerful compressors of biology. During training, the model will learn whatever representation it needs to in order to predict the hidden amino acid. Sequences inherently convey information on downstream biological properties, and learning this signal happens to be quite useful in minimizing loss. Deeper understanding emerges out of necessity. What's really exciting is that we can then use these unsupervised models + representations to learn more about unknown biology. There are many unannotated sequences that structure/sequence alignments cannot characterize. SAE features provide interpretable and semantically rich clues into the true nature of a protein when traditional methods fail. @salcandido said it best: most of the time we use mech interp to learn more about language models. Here, we use mech interp to learn more about biology. How poetic that techniques for better understanding ā€œalien intelligenceā€ could be used to better understand our own.
Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology. The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics. We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity. We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures. ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences. A world model of protein biology emerges through language modeling. We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins. The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science. This understanding emerges without prior knowledge, just from language modeling of protein sequences. Language models are becoming a powerful substrate to understand and program biology. The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders. I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.
4
28
1
161
13,105
Neil Thomas retweeted
I'm so excited to show the world what we've been working on the for the past months!! I'm going to highlight some of the fun results from this paper that I find particularly exciting.
Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology. The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics. We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity. We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures. ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences. A world model of protein biology emerges through language modeling. We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins. The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science. This understanding emerges without prior knowledge, just from language modeling of protein sequences. Language models are becoming a powerful substrate to understand and program biology. The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders. I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.
16
32
9
220
76,454