prompt surfer retweeted
This hole unlocks a new wave of companies, which build deep domain expertise to solve a vertical problem with expert quality automation !
Replying to @sama
this works amazingly well; it has been one of my biggest surprises of 2024.
excited to see what people build!
prompt surfer retweeted
As we continue to explore new post-training techniques, today we're releasing Llama 3.3 — a new open source model that delivers leading performance and quality across text-based use cases such as synthetic data generation at a fraction of the inference cost.
prompt surfer retweeted
Introducing: Runner H 🏃🤖
After raising a $200m seed round, @hcompany_ai just dropped information about Runner H.
It's a specialized 3B model that can accomplish real-world tasks for you.
Details and WILD demos below 🧵
Has Pre-Training Hit a Wall? A Closer Look at the Debate
There’s a growing narrative in the AI community that pre-training has reached its limits. Proponents of this idea argue that we’ve extracted most of the value from large-scale pre-training, and that future gains will require innovations in scaling test-time compute rather than in the pre-training phase itself. OpenAI’s O1 Preview, for example, is cited as a model demonstrating how allocating more compute at inference time can enhance capabilities without further scaling dataset size or training resources. The claim is that these methods are necessary because we’ve hit a wall in improving models through traditional pre-training.
I disagree.
The Untapped Potential of Pre-Training
The idea that pre-training has "hit a wall" overlooks a fundamental challenge: we haven’t yet trained models on the best possible data humanity has to offer. Much of the world’s most valuable information remains locked behind barriers of intellectual property (IP), copyright, and legal restrictions. These constraints limit the diversity and richness of training datasets, leaving models with only a partial representation of the knowledge they could potentially learn from.
Furthermore, even when such data is theoretically accessible, the cost of acquiring and curating rare, high-quality data is astronomically high. As a result, pre-training efforts are often limited to data that is widely available or less expensive to obtain. This creates a situation where the perceived plateau in pre-training performance may be less about the methodology itself and more about the scarcity of the right kind of data.
The Cost of Rare Data
The rarer and more specialized the data, the greater the expense. This economic reality means that much of the high-value, nuanced data necessary for pushing pre-trained models to the next level remains unused. Until these barriers are addressed, it’s premature to declare that pre-training has hit its limits.
Scaling Test-Time Compute: Complementary, Not Replacement
The move towards scaling test-time compute is an exciting innovation, and methods like O1 Preview demonstrate how inference-time improvements can lead to better model performance. These approaches enable models to simulate human-like reasoning by leveraging additional compute resources during inference, which is a significant step forward.
However, this doesn’t mean pre-training has reached its endpoint. Test-time compute and pre-training are not mutually exclusive. Instead, they should be seen as complementary strategies. Better pre-training, fueled by more comprehensive datasets, would enhance the capabilities of models even before test-time compute is applied. Conversely, stronger pre-training reduces the dependency on massive inference compute, creating a more balanced and efficient AI system.
Breaking the Data Barrier
If we want to continue advancing AI, we need to focus on breaking the data barrier. This involves addressing legal and economic hurdles to unlock high-quality datasets while ensuring ethical and responsible use. Innovations in data acquisition, such as better synthetic data generation or collaborative data-sharing frameworks, could also play a pivotal role.
The Future of Pre-Training
The belief that pre-training has hit a wall is shortsighted. We’re not at the limits of what pre-training can achieve; we’re at the limits of what our current access to data allows us to achieve. By overcoming the data scarcity challenge, we can unlock the next wave of AI progress. Test-time compute is a powerful tool, but it should enhance, not replace, the foundational role of pre-training in building smarter, more capable models.
Let’s not close the book on pre-training just yet. Instead, let’s work to ensure that AI has the data it needs to realize its full potential.
34
prompt surfer retweeted
Solar generates more electricity than nuclear in Texas.
Next year, coal.
In a few years, wind - and then gas.
prompt surfer retweeted
It's interesting to see how the various model families have evolved their code editing skill over time.
aider.chat/docs/leaderboards…
No amount of the Trump administration's fuckups will slow the global exponential progress we see in the price performance of computation and renewable energy. A clean energy universal high income future is near if we allow it to exist.
32
It was a waste of an experiment to not train the doctors on effective use of GPT-4. I'm certain there would've been different results had they been trained, had a month to use GPT-4, and do this test. A.I. Chatbots Defeated Doctors at Diagnosing Illness nytimes.com/2024/11/17/healt…
58
Atheism is as bold of a statement of knowing what is actually going on as being religious. I prefer being agnostic because it acknowledges, like science does, that we simply don't know.
Admitting you don't know takes courage. Having children isn't exclusive to religion. And just because you leave religion doesn't mean you stop caring about the world.
I won't believe in something simply because it keeps me from being sad. I believe in searching for truth which is what @xai is all about, right?
Atheism left an empty space
Secular religion took its place
But left the people in despair
Childless hedonism sans care
Maybe religion’s not so bad
To keep you from being sad
1
141


















