clern 😳 (@_buggingface_) header
clern 😳 (@_buggingface_) profile picture

goofy goober

Brooklyn, NY
Joined May 2023
Photo by clern 😳 (@_buggingface_)Photo by clern 😳 (@_buggingface_)Photo by clern 😳 (@_buggingface_)Photo by clern 😳 (@_buggingface_)Photo by clern 😳 (@_buggingface_)Photo by clern 😳 (@_buggingface_)
He’s got that Je n sen quoi
Replying to @yunta_tsai
A petition is underway to reinstate Jensen and I’m signing it too
2
47
If you see three points on a plot, the best estimate of the number of points on your plot is three
This slide from the Anthropic IPO roadshow was a big factor in my decision to leave. It’s an insane plot and I’m surprised they made it public. It shows the compute efficiency of pretraining over time. The y-axis is a **log scale**: they are seeing continued superexponential algorithmic progress. Mythos was not a one-time deviation, but a point on a superexponential. They could already retrain Mythos with a fraction of the compute originally needed. It doesn’t strictly imply a software-only intelligence explosion- they have been spending more on R+D. But algorithmic improvements do not require ridiculous creative genius and are on the verge of being automated. You do the math.
1
7
119
clern 😳 retweeted
Datology Curation Studio is out today, along with the experiments I spent all summer running to test it. I'm excited to finally talk about the research engineering that went into them! A lot of it was new for us, and the whole team really came together to make it work. 🧵
Today, we’re excited to launch the Datology Curation Studio, customizable frontier data curation for every team training its own models. Data quality is the ultimate compute multiplier, and Curation Studio makes it easy for everyone.
1
10
3
36
1,454
i heard they even beat the best buggingface data
Today, we’re excited to launch the Datology Curation Studio, customizable frontier data curation for every team training its own models. Data quality is the ultimate compute multiplier, and Curation Studio makes it easy for everyone.
1
18
431
Kid 1: Why was 6 afraid of 7? Kid 2: 67!!!!!
38
It’s pumpkin spice baby formula season
2
67
when babies put their hand in their mouth it’s called a pacifist
5
69
Worth the trip to Sydney for
Accepted to NeurIPS! 🇦🇺
1
2
469
Found jd vance’s anon
I treat objects like women
3
98
Now we know what the B in $15B stands for
4
49
Im officially changing the name of Artificial Intelligence to Intelligence of America
3
44
clern 😳 retweeted
Jeffrey has been on a mission, investigating and developing local learning rules. We know that the brain can't do backprop - at least not exactly - so it would be amazing if we could figure out what it is doing instead.
Introducing PC-ALM, a local-learning alternative to backpropagation. Our method trains 1000-layer neural nets using only local dynamics, and without backprop. Blog: pub.sakana.ai/pc-alm/ Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly. How can a physical system, such as the brain, solve multilayer credit assignment without explicit use of backprop? We look for inspiration in two related fields: distributed optimization and NeuroAI. In NeuroAI, predictive coding asks each neuron activation to solve an energy-based inference problem instead of using a standard forward pass. That inference step can be implemented as energy-minimization dynamics on local prediction errors. This perspective -- each layer as a dynamical system -- has proven promising, but performance of predictive coding hasn't scaled well with depth. Credit signals at far ends of the network struggle to diffuse into internal layers. We turn to distributed optimization, generalizing predictive coding to use an augmented Lagrangian instead of energy. This motivation stems back to a classic 1988 paper by LeCun, showing that the Lagrange multipliers of a deep network can be identified with gradients of a supervised loss. The augmented Lagrangian then bridges LeCun's perspective to the standard predictive coding that is used in NeuroAI. We find that this new perspective yields a natural PC-like alternative to backpropagation, resulting in a method we call PC-ALM. PC-ALM differs from PC in that it introduces dual neurons (Lagrange multipliers) as part of the layer-local dynamics, resulting in each layer acting as a PI feedback control system to minimize local prediction errors. We find that PC-ALM is capable of propagating signals to seemingly arbitrary depth, especially in deep narrow networks where standard PC struggles to learn. Ultimately, our motivation here is to understand how distributed physical systems, such as the brain, can compute credit signals using only local coupling and local dynamics. PC-ALM may also inform deep learning in neuromorphic hardware, where dynamics are cheaper than on GPUs. Paper: arxiv.org/abs/2605.31022 Code: github.com/SakanaAI/pc-alm
5
64
7,331
sharing a new paper! Augmented Lagrangian Predictive Coding I think one of the coolest unsolved problems is how the brain does credit assignment. thread 🧵
Introducing PC-ALM, a local-learning alternative to backpropagation. Our method trains 1000-layer neural nets using only local dynamics, and without backprop. Blog: pub.sakana.ai/pc-alm/ Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly. How can a physical system, such as the brain, solve multilayer credit assignment without explicit use of backprop? We look for inspiration in two related fields: distributed optimization and NeuroAI. In NeuroAI, predictive coding asks each neuron activation to solve an energy-based inference problem instead of using a standard forward pass. That inference step can be implemented as energy-minimization dynamics on local prediction errors. This perspective -- each layer as a dynamical system -- has proven promising, but performance of predictive coding hasn't scaled well with depth. Credit signals at far ends of the network struggle to diffuse into internal layers. We turn to distributed optimization, generalizing predictive coding to use an augmented Lagrangian instead of energy. This motivation stems back to a classic 1988 paper by LeCun, showing that the Lagrange multipliers of a deep network can be identified with gradients of a supervised loss. The augmented Lagrangian then bridges LeCun's perspective to the standard predictive coding that is used in NeuroAI. We find that this new perspective yields a natural PC-like alternative to backpropagation, resulting in a method we call PC-ALM. PC-ALM differs from PC in that it introduces dual neurons (Lagrange multipliers) as part of the layer-local dynamics, resulting in each layer acting as a PI feedback control system to minimize local prediction errors. We find that PC-ALM is capable of propagating signals to seemingly arbitrary depth, especially in deep narrow networks where standard PC struggles to learn. Ultimately, our motivation here is to understand how distributed physical systems, such as the brain, can compute credit signals using only local coupling and local dynamics. PC-ALM may also inform deep learning in neuromorphic hardware, where dynamics are cheaper than on GPUs. Paper: arxiv.org/abs/2605.31022 Code: github.com/SakanaAI/pc-alm
11
40
9
315
34,420
Nolnah’s razor
in love and war and silicon valley, never attribute to stupidity that which is adequately explained by malice
3
48
This post blew up in finite time. I’m Navier stoked!
I lowkey think Google might have solved Navier Stokes
1
3
158
Would’ve been $12T if named buggingface 😳
this is insane, if they named their company ☺️ back in 2016 they would only be getting acquired for 978 million
1
4
527
$2.5B run rate
currently selling one Microduck every 5 seconds
2
86
It’s me
This quoted post is unavailable.
1
3
107