@NuclearManD
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- Web
Account-level information from X, not a live location or the device used for a specific post.
Self-taught SWE. Anti-hype crypto user. Amateur linguist. Making LLMs more rational and predictable. Follow for huggingface pushes and research updates.
Brevard County, Florida
Joined June 2026
- Tweets717
- Following270
- Followers98
- Likes2.2K
Pinned Tweet
Working on a new transformer design which has dedicated logic targeting LLM weak points.
The model I'm training has 2.4B parameters and will have a free public API when it's ready.
I'm specifically targeting mathematics and expanded long-context memory.
Dylan Brophy retweeted
I think AI is going to do two completely opposite things to people: unlock an extraordinary amount of human potential and produce an extraordinary amount of resentment.
Think about how many people go through life never discovering what they're actually capable of.
The kid fascinated by physics who never had a decent teacher. The guy who's spent years dreaming about starting a business but can't code. The person who has a hundred ideas but neither the money nor the connections to pursue them.
Now imagine giving all of them access to a brilliant tutor, a programmer, a research assistant and a creative partner.
Imagine the amount of human potential we could unlock over the next twenty years.
I find that ridiculously exciting.
But there's another side to this.
For years, you could tell yourself you'd be incredibly successful if only you had the right opportunity. You could blame the system, your circumstances or the people who never recognised your genius.
Now you can actually try.
You spend a weekend building the business you've been talking about for five years. You launch it, show it to people, and absolutely nobody gives a crap.
Suddenly, your imagined potential has to meet reality.
And I think this is where people will split.
Some will embrace it. They'll accept that they're beginners, figure out what went wrong and keep improving. Give millions of people that opportunity, and imagine what humanity could become.
Others will become increasingly resentful. They'll watch people around them build things they thought were impossible, and every success will remind them of something they haven't attempted.
It's psychologically much easier to believe the game is rigged than to discover you're not as exceptional as you imagined.
AI will threaten some people's excuses before it threatens their jobs.
I suspect there's going to be an enormous psychological divide between those who see these tools as an opportunity and those who experience them as a threat to their identity.
We might be about to discover just how much human potential has been held back by circumstance, and how much has been hiding behind excuses. And I'm incredibly optimistic about the people willing to find out.
It's insane we've built a society that worships technology.
Technology is incredible but worshiping it and its proponents will not end well.
Transformer blew up by my house today. Good thing I have a UPS, training was not interrupted.
You could hear the electric arcs from a long distance.
Figured out how to speed up the training about 3x for now, will slow again when I need to expand what it learns (context, math)
Currently LLMs can't learn to abstract patterns between different concepts unless:
- The concepts are mentioned together in the training data
- The pattern can be learned in the same layer/region of the LLM
I have a crude but possibly useful solution to this.
I call this "parameter boosting", and I haven't tested it yet, but here's the idea:
- Create a bank of a large number of small neural networks (maybe 32768 networks of 64k parameters each, two or three layers, something like this)
- Each network gets an embedding assigned to it
- Each layer of the LLM gets a system to route the residual into one or more of these small networks
- The LLM layer's FFN accepts the result and the embedding as input
It is similar to MoE, but with three important differences:
- These small networks ("patterns") are shared across all layers of the LLM, so the same pattern can be learned once and used in multiple parts of the LLM on different data
- These 'patterns' have an embedding to identify them, allowing LLMs to describe what circuit it is using when responding, thus allowing it to describe patterns which were not in the original training data.
- The routing would need to be different from MoE for it to effectively work. This part I haven't figured out yet.
I call it "parameter boosting" because it allows the model to abstract the same behavior needed in multiple parts of the model to one network, thus possibly increasing the usefulness of each parameter.
This is just an idea I've been thinking about, all theoretical, need to test it still.
Training progress: After ~81 hours of training so far, some simple grammar patterns are showing up. It's still early, and I lost a lot of time due to the bug I previously described.
I put together about 190M tokens of training data, may need more. I might need to rent a GPU.
So far I've done about 6200 steps, or about 101M tokens. Loss is 3.4-3.8 on this new dataset, which has a lot more text (less code), previously I got down to 1.4 or so. These numbers are confusing though, as the loss isn't purely cross-entropy on this model.
Other tests I've done indicate that the architecture can natively learn math without memorizing patterns in text, so my hope is that this model will really kill it in math particularly, freeing up some parameters to focus on other tasks.
Previously I had trained various broken models for maybe 100 or more hours... so much lost training time... but it is working this time! Just slowly.
Dylan Brophy retweeted
UNPOPULAR OPINION: We built robots to do the dangerous jobs and instead pointed them at the easy ones.
Why isn't AI down in the mines, or on offshore oil rigs, or handling industrial chemicals, or pulling plastic out of the ocean?
Instead it's drafting emails and building slide decks for people who already have desk jobs with air conditioning.
We automated the part of work that was never going to kill anyone, and left the actually dangerous stuff for humans to keep doing.
What exactly are we optimizing for here?
Progress update on the model I'm training: It seems to be able to learn math as intended alongside text and token patterns, but it takes a while to train with limited compute😅
I found a critical bug causing large numbers (+/- thousands) to flow into the residual. This prevented the model from training by drowning out the gradients of the transformer layers.
Now that it's fixed, it's finally learning. My architecture seems to learn a bit slower than transformers, the hope is that it will learn better eventually while using less parameters, thus making it cheaper to run.
This model will be focused on better mathematics, spatial comprehension, and long-context memory on a smaller model.🧮 When I'm done I'll provide a free open API and chat interface for y'all to try it out with. 🚀
I also plan to test another architecture separately to allow LLMs to learn on the fly, have infinite context, and learn by reading instead of via training data. This will hopefully allow a much smaller and cheaper model to have higher intelligence as well.
Stay tuned for updates!
"the fact remains that lying at the foundations of every worthwhile computer system is a mental model of its intended operation" This.
such bollocks. bro throws out some half baked idea that we're all writing code that doesn't matter and we can just yolo our way with a code fountain and abandon our responsibilities.
sometimes i do wonder what the hell people actually write code for.
bottom line is 99% of those of us who actually ship products are supporting profitable businesses who are heavily reliant on quality predictable and maintainble outcomes.
sure, if you're sat in your bathrobe producing the foundations for your next book, or writing mickey mouse code to put in your next paper, or if you're a data usher in a close-to-zero users SaaS where the only effect of errors or downtime is "oh my users can't produce pretty pictures" then none of the code you're writing matters anyway.
the fact remains that lying at the foundations of every worthwhile computer system is a mental model of its intended operation, including a deep enough understanding to support the ongoing improvement and maintenance to ensure a solid foundation upon which a profitable business sits.
i fear that we are going to learn some very hard lessons if the academics and toy-builders get enough air to influence the direction of our industry.
Dylan Brophy retweeted
We have become “disembodied” in our philosophy to the point that we believe a statistical math model is equivalent to a biological body.
It’s really disconcerting.
Dylan Brophy retweeted
je fais partie de cette génération 90 qui a littéralement le cul entre deux chaises
on est nés sur la couture exacte de deux époques: trop tard pour appartenir au monde des cassettes et des cabines téléphoniques, trop tôt pour être de purs enfants des écrans
on garde la mémoire musculaire du monde d'avant tout en manipulant couramment le code du monde d'après et en ce sens j’ai l’impression qu’onest la dernière génération à savoir à quoi ressemble le silence avant le bruit du réseau et la première à devoir naviguer dans son effondrement
je voulais écrire ces quelques lignes qui sûrement vont parler à d’autres camarades de ma génération mais je ne vous cache pas que pour moi cet entre-deux est un séisme silencieux perpétuel car je vis l'agonie d'un monde qu'on a connu et l'accouchement d'un autre qu'on ne nous a pas appris à habiter
cependant, je crois qu’au fond cette solitude est peut être ausis un privilège stratégique car elle elle nous donne potentiellement la distance nécessaire pour ne pas vénérer les ruines du passé, ni subir aveuglément les architectures de l'avenir
voilà, voilà
AI allows me to do a year or more of work in days, I just don't know if it's *good* work. But for now, it is good enough.