@BetaTomorrow
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- United States Android App
Account-level information from X, not a live location or the device used for a specific post.
mathematics Thief & Chef "through the window of differential equations, mathematics sees the light in the real world" / "通过微分方程的窗子,数学家看到现实世界的光" (Jiang Zehan)
Seattle
Joined June 2008
- Tweets14.6K
- Following645
- Followers4.2K
- Likes6.1K
Pinned Tweet
“This Is Not How Mathematicians Are Trained”
数学家不是这样训练的
1, "Solving the forward problem (positive time) and the inverse problem (negative time) together has always been a desire of mathematicians, but they’ve never known where to begin. Neural networks, however, tackle this problem naturally" (将正时间的正问题与负时间的反问题同时求解，一直是数学家的梦想，但他们始终不知道从何入手。而神经网络却能自然地处理这个问题)
2. “Variables, coefficients, even coordinates, are changing. Everything is in flux. This is not how mathematicians are trained. It would be impossible for mathematicians to come up with such a design” (变量、系数，甚至坐标都在变化，一切都处于变动之中。数学家不是这样的训练。这样的设计，不可能出自数学家之手)
3. “Mathematicians tread carefully around composite functions with more than two layers, wary of the many pitfalls, yet neural networks solve them effortlessly, almost nonchalantly.” (数学家在处理超过两层的复合函数时格外谨慎，警惕其中诸多陷阱, 而神经网络却几乎漫不经心地轻松应对)
4. “Neural networks have stacked covers, mathematically speaking, whereas the Numerical Manifold Method typically uses only 3 to 4. In contrast, neural networks stack hundreds or even thousands of such covers. I never imagined anyone would take it that far”. (神经网络在数学上拥有堆叠的覆盖层，而数值流形通常只有三到四层覆盖。相比之下，神经网络则堆叠了上百乃至上千层。我从未想过有人会将其推进到这种程度).
That was what Gen-Hua Shi (石根华) told me after returning from a two-week vacation in early June 2024..
see rest of the story, click the link
open.substack.com/pub/deepma…
So happy to see this... I have been repeatedly saying history is repeating itself... The reason I said that is because it has historical precedents.
On 08.29, I wrote in "When Mathematical Beauty Becomes Useful in AI" that "Some even believed that advances in discretization and computational power might make much of traditional mathematical analysis less important."
nitter.cf/BetaTomorrow/status/20…
Mathematician Alain Connes, on the motivation for creating the Millenium Prize challenges:
'In contrast to Hilbert’s speech — and reflecting concern in some quarters that the CMI might be attempting a similar task — those responsible for the prize emphasize that they have no desire to determine the direction in which mathematics moves forward.
Rather, they are trying to recreate excitement about the activity of mathematics, both in the general public and among school students. “Sometimes people have the wrong idea of maths and think that it will be overtaken by computers,” says Connes. “The seven problems, each selected by top specialists in their respective fields, are totally inaccessible to computers.”'
Computers have come a long way.
Source: David Dickson. "Mathematicians chase the seven million-dollar proofs". Nature; London Vol. 405, Iss. 6785, (May 25, 2000)
"Deep Manifold Part 2: Neural Network Mathematics" arxiv.org/abs/2512.06563
Zienkiewicz Moment, History Repeating Itself ?
nitter.cf/BetaTomorrow/status/20…
AI is an inverse problem. Treating AI as a forward problem is a world-class, once-in-a-century mistake.
AI是反问题，大家都把AI作为正向问题来做，这是世界级、世纪级的错误。
“We started with the geometry”
but what exactly is the geometry here? The article never defines the geometric object, its coordinates, its structure, or what geometric quantities are being measured. Without such a definition, “geometry” is doing more rhetorical work than mathematical work.
meet @mostik_ai!
what happens when you put 12 PhDs in one room for four months? first place on the ARC-AGI leaderboard, which I can't say much about while the competition is still running. and this, which I can.
everyone's arguing about whether open models will catch up to frontier models. we think it's the wrong question. here's the one we pose: why does a frontier model have to generate your answer at all, when the only thing you need from it is the reasoning?
we do this by enabling models to communicate in latent space. through our protocol, hidden states pass straight from a frontier model into a small one running on your infrastructure -- no text between them, and neither model is fine-tuned. two models from different families, sharing reasoning, both left untouched.
how do we know it works? we tested it on a setup where a 753B model reads the problem, and a 4B edge-class model writes the answer. with this approach, we get results 80% as accurate as the frontier model, but at 20x faster performance.
we're committed to preventing frontier model lock-in and are already partnering with inference providers to accelerate open-weight adoption. we've done this between 15 of us, in four months, 12 PhDs and a Fields medalist, backed by @generalcatalyst
WIRED has the first external account of the company and the work: wired.com/story/russian-star…
full writeup, the setup, and all the numbers: mostik.ai/read-more
#DeepManifoldInterpretation
Paper: Deep Equilibrium Models
Author: Shaojie Bai (@shaojieb), Zico Kolter (@zicokolter) and Vladlen Koltun
The loop is central to the DEQ design. The authors deliberately formulate the network as a discrete dynamical system, repeatedly applying the same learned transformation to the hidden state until it reaches an equilibrium, or fixed point.
This makes depth a form of recurrence: the same map is reused while the activation evolves from one loop to the next.
DEQ then replaces explicit unrolling of many such loops with a root-finding procedure that directly solves for the state where the residual vanishes.
In this sense, the equilibrium is the computational endpoint of the loop, and the fixed point becomes more fundamental than the individual layer.
Deep Manifold starts at a more fundamental level: model prediction is written as x=f(x)x=f(x), a fixed-point relation, and then transferred into a Lagrangian formulation.
This not only makes the problem computable, but also reveals the dynamic nature of neural networks as residual-driven, iterative systems shaped by changing boundary conditions.
DEQ makes this loop explicit by architectural design; Deep Manifold treats the fixed-point and iterative structure as part of the neural-network equation itself.
The concept of Looped Transformers reminds me of the literature on Equilibrium Models (e.g. arxiv.org/abs/1909.01377) which studied to some extent the limit of looping. It is still unclear in which cases such models could be better than vanilla architectures (no looping).
#DeepManifoldInterpretation
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Author: Shaowen Wang et al.
From the Deep Manifold perspective, the second loop can be understood as a residual-refinement step.
The first traversal performs the primary transformation, but because learning and numerical transformation are dynamic, that transformation also leaves behind residual structure and creates new secondary relationships in the updated activation state.
The second loop does not restart from the original input; it operates directly on this already-transformed state. In this sense, it can be viewed as attempting to reduce the residual left by the first loop, while also accounting for part of the secondary numerical structure created by the first transformation itself.
This makes looping more than added depth: it is a repeated numerical correction over the consequences of the model’s own previous computation.
Can Looped Transformers still help when FLOPs, parameters, and KV cache are all matched?
We introduce SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers.
The answer is yes. And the advantage grows with scale. 🧵 1/8
Paper: arxiv.org/abs/2609.01343
#DeepManifoldInterpretation
Paper: Looped Transformers as Programmable Computers
Author: Angeliki Giannou (@AngelikiGiannou) et al.
The paper offers a particularly clear example of the Deep Manifold view that neural-network inference is an iterated integral rather than merely next-token prediction.
The paper shows that a fixed Transformer with fixed weights can repeatedly process an evolving whole state, scratchpad, memory, and instructions, so computational depth comes from recurrent traversal of the same weight-defined manifold rather than from adding more layers or new weights.
In this reading, each loop carries the residual computational state forward, which is why recurrence effectively handles the residual’s residual.
The result is a clean separation between a fixed computational geometry and the progressive execution taking place within it.
At the same time, because the Transformer weights are manually constructed and never learned or updated, the paper demonstrates forward computation over an already-defined manifold, not the inverse problem of how training forms that manifold in the first place.
Looped transformers arxiv.org/abs/2301.13196 @DimitrisPapail
