@its___jayanti
iAccount based inIndia
About this account
- Account based in
- India
- Connected via
- India Android App
Account-level information from X, not a live location or the device used for a specific post.
ML + Kernel engineer @AMD. Making LLM inference actually fast on real hardware. llama.cpp contributor. NOVA paper on Winograd
Bangalore
Joined September 2025
- Tweets885
- Following164
- Followers62
- Likes106
Pinned Tweet
1/
I got offers from major Quant firms, compute and tech companies, and startups from the US in 2026. None came from job applications.
None came from interviews. I only cold emailed senior people
while alive:
desire = anticipate()
reward = acquire(desire)
if desire > reward:
feeling("empty")
desire = next()
Maybe contentment is learning to break the loop
128 gigs of unified memory. So many experiments to run. Scaling RL tops the list rn. Will share the insights along the way!
the interesting bit in the writeup is that fable wasn't handed this cipher
it was told to find an unsolved one and picked this
choosing which problem to spend your next hour on is a pretty useful thing to get right
Fable solved the Cyphral Distich (a 370 year old cypher). Super cool way to use Claude
vals.ai/blogs/fable-solves-c…
if a specialized engine finds a faster way to generate tokens and that ends up in vllm
that sounds like the project worked
not every repo needs to become a company with a discord server
my hot take is specialized inference engines are amateur projects to spend the spare tokens when @thsottiaux gives people one more reset😄
an inference engine is way more than just running a model on a certain hardware, it is an inference ecosystem.
I use this slide several times when I talk to people about vLLM. Models, hardwares, and inference techniques, all of them move quickly. And vLLM lies in the intersection to provide a unified interface to end-users and applications. vLLM is the inference ecosystem.
vLLM not only support current models and hardwares, but we are also working to support new models and hardwares coming in the next few months. It's an ecosystem people can trust and rely on.
Over the past 3 years, I have seen so many projects claiming to be better than vLLM in certain aspects, but in the end either their techniques are contributed to vLLM or they disappear. That's the power of ecosystem.
A specific example would be tilert github.com/tile-ai/TileRT , a megakernel inference engine dedicated for decode. They collaborate with vLLM by using vLLM prefill + tilert decode in vllm.ai/blog/2026-07-14-vllm… , as highlighted in newsletter.semianalysis.com/… from @SemiAnalysis_ .
the ram adds up neatly on the shopping list
the time spent waiting for the other laptop is the part i'd benchmark
show me one chat getting faster before showing me ten chats running at once
real-swe has a task where the agent has to fix invoice taxes without charging exempt customers
this is the annoying part of replacing engineers
the tax function can be perfectly written and still send the wrong person a bill
Introducing Real-SWE, a coding benchmark built from private company codebases.
We evaluate coding agents on real software engineering tasks that engineers performed, using private, out-of-distribution codebases.
These tasks come from real companies, examples include an app with 200K+ users, fintech platform processing 100K+ bank statements, enterprise sales tools, etc.
The results are quite surprising.
all the ai labs agreeing on something has somehow made me more nervous than all the ai labs disagreeing
rik arends built an explorer for a 2.5 million line codebase
we keep giving coding agents bigger context windows while the human reviewing their work gets a file tree and 40 open tabs
i'd like some of that tooling budget on the human side too
somewhere between the paper and this tweet we went from pruning networks trained on handwritten digits to declaring 96% of claude useless
finding a tiny network you can train is a result
knowing which weights to delete from claude is another result you still have to show
As someone who ships LLM systems in production, this scaling video is the closest thing to a "why 96% of Claude and GPT-5's weights are literally useless" explainer I've ever seen released for free.
Everyone thinks trillion-parameter models need every parameter. They don't. A 2019 pruning experiment proved you can delete 96% of a neural net's weights with zero performance loss - meaning most of Claude and GPT-5 is empty scaffolding around a tiny "winning lottery ticket" network doing all the real work.
Bookmark this 18-min video and watch tonight. Same lottery ticket math from 2019 MIT research, now the reason every AI lab wastes 90%+ of its Nvidia budget.
Readers added context they thought people might want to know
The 2019 LTH paper found winning tickets (typically 10-20% size) matching perf on small MNIST/CIFAR models only when retrained from specific inits. It does not prove 96% of trained trillion-param LLMs' weights are useless; IMP fails to scale without major changes.
arxiv.org/abs/1803.03635
arxiv.org/abs/1903.01611
openai reports ~130 billion output tokens on navier stokes
i'd love to know how many went into the idea that survived and how many went into finding out what doesn't work
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
the interesting bit in arm's new gpu is where the neural hardware sits
inside the shader core, sharing the gpu's caches and memory system
doing the ai math is only part of the job. moving frame data around can eat the savings
Rendering every pixel? Not anymore.
Neural Dawn combines Arm Neural Technology with the new Mali G2-Ultra NX, our best yet, to deliver fully ray-traced lighting and smoother gameplay with less graphics work.
Coming soon to mobile. 👇
Ensure your latest flagship mobile is powered by Mali G2-Ultra NX to experience richer visuals and new levels of effciency. okt.to/irRwS7
the interesting part of ripwire is giving the agent callers and tests alongside the code
i'd measure how often it misses an affected caller after a change
fewer tokens is nice. fewer missed dependencies would make me keep it
ripwire, from Red Hat Emerging Technologies, is a remarkably substantial new approach to giving coding agents repository context without embeddings, a vector database, an LLM indexer or a daemon. The zero-dependency C++23 binary parses 21 languages with Tree-sitter and builds a deterministic structural map of a codebase, ranking symbols for the task while attaching call relationships, complexity, git churn, change amplification and test coverage.
An agent can ask what matters for "incremental cache invalidation," for example, and receive the relevant symbols, their callers, likely blast radius and tests to run in a token-budgeted response instead of grepping and opening files repeatedly.
GitHub Repo: github.com/redhat-et/ripwire
the fun bit is how loops work
no bytecode saved anywhere. it jumps back and parses the source again every iteration
spending cpu time to save interpreter code gets this tiny python subset into 1024 bytes of c
Every good weekend includes coding a toy interpreter.
austinhenley.com/blog/python…
astra is killer for kernel perf engineering. once i told it where to look it turned the optimization in my head into a working speedup in one shot
BTW both Fable 5.1 and GPT 6 failed, overnight, on a problem that I solved in 5 minutes. I want to post what happened just to document why we're not quite on AGI yet, although getting closer every day.
This post will be a bit technical because details matter
So, last night Fable identified a memory leak on Bend2. A given long-running process would steadily grow in memory, until it crashed in just a few hours. That's the bug.
I asked both Fable and Astra to fix it overnight. Rules:
1. Do not increase the codebase size
2. Do not cause regressions in any benchmarks
Both failed. But why?
First, both identified the correct root cause.
(everything below is written by Fable)
Bend2's runtime has 16k GPU lanes. Each lane has its own private free lists, so alloc and free are 2 stores, no atomics. That's what makes it fast. The catch: when a lane frees a block that another lane allocated, the block lands on the freeing lane's list. If that lane never allocates as much as it frees, the block sits there forever. Nothing is lost, the memory is just parked on a list nobody uses. Both models found this. Both measured it: 40-85% of all frees cross lanes.
Then both spent the whole night trying to fix the allocator.
Return blocks to the owner lane: +124% slower. Slabs: 73x slower. Buddy: +7-21%. Flush at sync points: +3.5k tokens. A full allocator rewrite with lock-free chains: leak gone, but +1.9k tokens and matmul 78% slower on CUDA. Every design either broke the token cap or a benchmark. Neither model converged. They just kept building more machinery on the same premise.
The premise was wrong.
I asked for a histogram of where the parked memory actually was. 96% of it sat in the last 4096 lanes. That's not an allocator problem. That's the scheduler: it hands out work as a pure function of lane id. Lane 0 always gets the root of the tree. The same lanes always get the leaves. The join always runs on the last child to arrive, and the same lanes always arrive last. So every launch, memory flows in the same direction, from the same lanes to the same lanes, and the flow never reverses.
The allocator was fine. The lanes weren't interchangeable.
Fix: rotate lane identities. Each launch, physical lane p uses the free lists of lane (p + r) mod 16k, with r advancing by a golden-ratio step. Over a cycle every identity plays every role once, and the imbalance sums to zero. No allocator change, no atomics, nothing on the hot path. A handful of lines. Once I told Fable this, it implemented it, measured it, and ran the whole gate in an afternoon: 555MB → 23MB after 2000 launches, OOM pushed from ~7k launches to ~75M, zero cost on all 16 benchmarks.
So what was missing? Not capability. Both models can write a lock-free allocator overnight. What neither did was step back and ask why memory flowed in one direction at all. They accepted the asymmetry as a fact of the system and built increasingly complex machinery to undo it, instead of removing it. That's the gap: not "can it do the work", but "does it question the frame it's working in". Still, it did everything else, from the diagnosis to the census to the fix, once pointed. Closer than last year. Not there yet.