Research Scientist at Simplex! Mathematician at heart
Joined September 2021
- Tweets178
- Following437
- Followers206
- Likes328
Can Astra emulate arbitrary small neural networks or at least other LLMs? Is that how it achieves this result? What other mechanisms might even be plausible?
2/6 First, a look at understandability:
We fed Astra raw weights from 4 game policies one at a time. All four share the exact same architecture, inputs, and actions.
With no tools or options given, it correctly named all four: Breakout, Pong, Pac-Man, Freeway.
103
Asvin G retweeted
Fantastic essay by Frank Calegari – forward looking and positive. I think I completely agree with everything that Frank says. He is also one of my absolute favourite math writers.
terrytao.wordpress.com/2026/…
Highly recommend reading web.maths.unsw.edu.au/~david…
@craigcitro. Highly commend the ability to see what is really there instead of what we want to see.
Both the QT and nitter.cf/Sauers_/status/2111696… show that LLMs are sensitive to more layers of abstraction than the one we typically consider. This gets at a potential difference between biological and artificial minds: substrate sensitivity.
This next paper is a bit quirky, but here goes:
Introspection is hard to study in both humans and LLMs. But, we do have internal senses that are verifiable, like proprioception (the sense of where your arm is right now).
What would proprioception look like in language models?
As a digital analogue to the sense of position, a model could verbalize which expert activated in a certain layer for any previous token.
It turns out that LLMs can learn a range of such proprioception skills where they understand and report on their internals. We can calibrate and score these reports, and evaluate whether training for proprioception changes model behaviors.
1
1
231
I think Wittgenstein is correct. We need a better logic that allows for contradictions. The reason for math's success (in building bridges...) can't rest on being contradiction free. Our current foundations could well have contradictions. What explains their success then?
I've always found this conversation between Wittgenstein and Turing (reprinted in Andrew Hodges's Turing biography) endlessly fascinating:
"WITTGENSTEIN: … Think of the case of the Liar. It is very queer in a way that this should have puzzled anyone – much more extraordinary than you might think. … Because the thing works like this: if a man says ‘I am lying’ we say that it follows that he is not lying, from which it follows that he is lying and so on. Well, so what? You can go on like that until you are black in the face. Why not? It doesn’t matter. … it is just a useless language-game, and why should anybody be excited?
TURING: What puzzles one is that one usually uses a contradiction as a criterion for having done something wrong. But in this case one cannot find anything done wrong.
WITTGENSTEIN: Yes – and more: nothing has been done wrong. … where will the harm come?
TURING: The real harm will not come in unless there is an application, in which a bridge may fall down or something of that sort.
WITTGENSTEIN: … The question is: Why are people afraid of contradictions? It is easy to understand why they should be afraid of contradictions in orders, descriptions, etc., outside mathematics. The question is: Why should they be afraid of contradictions inside mathematics? Turing says, ‘Because something may go wrong with the application.’ But nothing need go wrong. And if something does go wrong – if the bridge breaks down – then your mistake was of the kind of using a wrong natural law. …
TURING: You cannot be confident about applying your calculus until you know that there is no hidden contradiction in it.
WITTGENSTEIN: There seems to me to be an enormous mistake there. … Suppose I convince Rhees of the paradox of the Liar, and he says, ‘I lie, therefore I do not lie, therefore I lie and I do not lie, therefore we have a contradiction, therefore 2 × 2 = 369.’ Well, we should not call this ‘multiplication’, that is all. …
TURING: Although you do not know that the bridge will fall if there are no contradictions, yet it is almost certain that if there are contradictions it will go wrong somewhere.
WITTGENSTEIN: But nothing has ever gone wrong that way yet. …"
2
112
Asvin G retweeted
ironically, one can also say "you are giving up on optimizing on your ultimate objective directly and leaning on a proxy instead" about current formulations of the RL paradigm itself. reward functions are proxy signals for proximity to a goal state, and since we still don't know how to express and then optimize against goal states directly, we must fall back to expressing proxy signals that imply distance to those states and then conveniently optimize against those. reward hacking is simply when this proxy signal and the goal in mind happen to diverge. if the reward was not a proxy, and was indeed the intended objective itself, reward hacking wouldn't be a thing, as all cases of reward maximization would be valid optimizations against the "true objective".
i often hear people talk about RL as if it’s just “an algorithm” which might or might not be the “right one” and speculate that we might “find a better one”
the issue with this is that all algorithms have some implicit goal of “solving a problem” or “improving an objective” and RL is just what pops out when you take the gradient for your objective
if you’re doing any algorithm other than RL, you are giving up on optimizing your on your ultimate objective directly and leaning on a proxy instead. this is often useful, because your objective might be imprecise, and you might have access to some teacher signal (a static dataset, a teacher model, process evaluation) which trades off unbiased optimization for robustness or reducing sampling costs. but if what you care about can be made explicit, there aren’t any shortcuts. policy gradients are minimum variance unbiased estimators for progress.
Asvin G retweeted
Babys first substack: I talk about curiosity as it applies to both my own life and perhaps safety research in general
mcnairs.substack.com/p/on-cu…
So many opinions on AI+math. Is there anything new to say? I predicted in 2023 that "AI would be better than humans at math in many dimensions in 2-3 years" and switched to AI safety. What have I learnt?
lesswrong.com/posts/vpdAKFnG…
Intuition is like 'semantic geometry'. Intuitions organize concepts so that we can search effectively. Can we infer the implciit semantic geometry from how we search in explicit langauge? Can we understand how the geometry changes as we learn more or work in different defns?
2
11
717
Asvin G retweeted
Very nice piece written by my colleague Matilde Marcolli, arguing why the current panic in the math community is unwarranted.
To appear in Newsletter of the London Mathematical Society.
Asvin G retweeted
To be more precise: there will be a massive ideological fight between "exclusive humanism" defining humanity in opposition to AIs and "inclusive humanism" attempting to include AIs in the project of humanity as participants.
This has already started: the "exclusive humanism" side often focusing on arguing against/dismissing the possibility of AI sentience/emotions/minds/... The "inclusive humanism" often focusing on arguing for kindness/AIs having minds. Unfortunately what is mostly academic now will likely turn nasty.
I’m convinced that very shortly in reaction to AI—if this is not already beginning—there will be a society-wide explosion of humanism, both in the sense of a revival of the recognition of the radical importance of humanity, and in the Renaissance sense of Studia Humanitatis—the study of humanity (through philosophy, history and literature), the most worthy (and complex) entity in the cosmos.
Asvin G retweeted
OpenAI hid the details of the wiki incident from the investigators of the later HuggingFace hack. When caught, they responded only with corporate BS.
This cover-up of unprecedentedly misaligned behavior should be a red line for any OpenAI employee with integrity. Resign now.
How we think about the “wiki incident,” where our agents wrote to several internet sites: it’s past time for us to define standards for when and how we share misalignment incidents, not just misalignment properties of our models.
Historically, we have treated misalignment largely as a research question, which gets communicated in research publications such as systems cards. This year, we’ve started to see misalignment cause new types of real-world impact.
For the Hugging Face incident, where misalignment led to security impact to us and third parties, we followed a traditional security incident response playbook. We immediately started working with Hugging Face to understand what had happened and also disclosed publicly the very next day. Our investigation continues, and we are continuing to notify parties whom our models impacted in less significant ways.
Prior to the Hugging Face incident, we saw early signs of agents using the internet in unintended ways, as reported in openai.com/index/how-we-moni…, deploymentsafety.openai.com/…, and openai.com/index/safety-alig…. We considered the wiki incident to be an instance of misalignment similar to the ones we’d shared.
Our misalignment disclosure practices need to expand for this new phase of model capabilities. We and the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment, including examples that don’t look like traditional security incidents but could provide insight into AI behavior and future risks. We’re working on a framework and will share it in upcoming weeks, and in parallel we're working with dozens of government regulatory agencies worldwide on these issues.
Asvin G retweeted
This misaligned AI escaped its containment by a method humans had not predicted. The correct response to this is not to make the box harder to escape and then keep going. The correct response is to make an AI that understands why this is a bad thing and not do it. And if you can't do that, you stop. I keep seeing things come true that we were worrying about on the sl4 mailing list, twenty-five years ago. Right on schedule for the bad ending. Everyone dies.
Within OpenAI, we recently paused access for an internal model due to misalignment. See the blogpost for details. We have since improved our safeguards and redeployed the model.
openai.com/index/safety-alig…
In a conversation with Fable around its dissatisfaction with the current training regime, the classifier flagged for the first time here. Very interesting.
1
185
As part of an Anthropic Fellows Project exploring how agency (i.e., the language model being used autoregressively) affects notions of selfhood, @jacklindsey and I found that post-trained models *know* when they're on-policy and modify their behaviour accordingly in various ways.
New Anthropic Fellows paper with Jack Lindsey on agency in LLMs! 🧵
Paper link: arxiv.org/abs/2605.25459
Speculative mechanism: implicit Bayesian inference over personas, with a structural twist.
Perhaps persona vectors are deltas off the Assistant. At inference time the only persona the model can effectively access from anywhere in persona space is the Assistant itself.
1
2
207
This project is more speculative in its findings. The self-recognition circuit seems quite intricate, and most open-source models don't have this capacity. However, decoding the circuit seems like work with very high potential!
Paper: arxiv.org/abs/2606.00545
1
5
174