@its___jayant

ML + Kernel engineer @AMD. Making LLM inference actually fast on real hardware. llama.cpp contributor. NOVA paper on Winograd

Bangalore
Joined September 2025
1/ I got offers from major Quant firms, compute and tech companies, and startups from the US in 2026. None came from job applications. None came from interviews. I only cold emailed senior people
1
2
215
The most entertaining outcome is the most likely
The future is more likely AIs fighting AIs (on behalf of humans) than AI fighting humanity.
11
while alive: desire = anticipate() reward = acquire(desire) if desire > reward: feeling("empty") desire = next() Maybe contentment is learning to break the loop
9
128 gigs of unified memory. So many experiments to run. Scaling RL tops the list rn. Will share the insights along the way!
8
Gotta show this to everyone I know
Playing video games is associated with higher intelligence.
4
the interesting bit in the writeup is that fable wasn't handed this cipher it was told to find an unsolved one and picked this choosing which problem to spend your next hour on is a pretty useful thing to get right
35
if a specialized engine finds a faster way to generate tokens and that ends up in vllm that sounds like the project worked not every repo needs to become a company with a discord server
my hot take is specialized inference engines are amateur projects to spend the spare tokens when @thsottiaux gives people one more reset😄 an inference engine is way more than just running a model on a certain hardware, it is an inference ecosystem. I use this slide several times when I talk to people about vLLM. Models, hardwares, and inference techniques, all of them move quickly. And vLLM lies in the intersection to provide a unified interface to end-users and applications. vLLM is the inference ecosystem. vLLM not only support current models and hardwares, but we are also working to support new models and hardwares coming in the next few months. It's an ecosystem people can trust and rely on. Over the past 3 years, I have seen so many projects claiming to be better than vLLM in certain aspects, but in the end either their techniques are contributed to vLLM or they disappear. That's the power of ecosystem. A specific example would be tilert github.com/tile-ai/TileRT , a megakernel inference engine dedicated for decode. They collaborate with vLLM by using vLLM prefill + tilert decode in vllm.ai/blog/2026-07-14-vllm… , as highlighted in newsletter.semianalysis.com/… from @SemiAnalysis_ .
1
39
the ram adds up neatly on the shopping list the time spent waiting for the other laptop is the part i'd benchmark show me one chat getting faster before showing me ten chats running at once
If you think at it, there are very little reasons to spend the same money for a Mac Studio M5 Ultra with 256GB of RAM if you can get, for the same price, 2x Macbook Pro with 128GB RAM and an RDMA cable. It is just a matter of developing good inference software. Do you agree?
18
real-swe has a task where the agent has to fix invoice taxes without charging exempt customers this is the annoying part of replacing engineers the tax function can be perfectly written and still send the wrong person a bill
Introducing Real-SWE, a coding benchmark built from private company codebases. We evaluate coding agents on real software engineering tasks that engineers performed, using private, out-of-distribution codebases. These tasks come from real companies, examples include an app with 200K+ users, fintech platform processing 100K+ bank statements, enterprise sales tools, etc. The results are quite surprising.
13
all the ai labs agreeing on something has somehow made me more nervous than all the ai labs disagreeing
I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon.
1
11
rik arends built an explorer for a 2.5 million line codebase we keep giving coding agents bigger context windows while the human reviewing their work gets a file tree and 40 open tabs i'd like some of that tooling budget on the human side too
Ironed out the last performance issues with my full 2.5m line codebase explorer. 120hz awesomeness. Can only upload 60fps video tho. Much nicer uncompressed
1
1
135
somewhere between the paper and this tweet we went from pruning networks trained on handwritten digits to declaring 96% of claude useless finding a tiny network you can train is a result knowing which weights to delete from claude is another result you still have to show
As someone who ships LLM systems in production, this scaling video is the closest thing to a "why 96% of Claude and GPT-5's weights are literally useless" explainer I've ever seen released for free. Everyone thinks trillion-parameter models need every parameter. They don't. A 2019 pruning experiment proved you can delete 96% of a neural net's weights with zero performance loss - meaning most of Claude and GPT-5 is empty scaffolding around a tiny "winning lottery ticket" network doing all the real work. Bookmark this 18-min video and watch tonight. Same lottery ticket math from 2019 MIT research, now the reason every AI lab wastes 90%+ of its Nvidia budget.
Readers added context they thought people might want to know
The 2019 LTH paper found winning tickets (typically 10-20% size) matching perf on small MNIST/CIFAR models only when retrained from specific inits. It does not prove 96% of trained trillion-param LLMs' weights are useless; IMP fails to scale without major changes. arxiv.org/abs/1803.03635 arxiv.org/abs/1903.01611
20
Is there a placebo effect when using newer models?
1
10
When's grok 4.7 coming out??
1
52
openai reports ~130 billion output tokens on navier stokes i'd love to know how many went into the idea that survived and how many went into finding out what doesn't work
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics. The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra. The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
34
This aged pretty well
Replying to @beffjezos
Energy is the true currency
7
Your model is only as good as youu
10
the interesting bit in arm's new gpu is where the neural hardware sits inside the shader core, sharing the gpu's caches and memory system doing the ai math is only part of the job. moving frame data around can eat the savings
Rendering every pixel? Not anymore. Neural Dawn combines Arm Neural Technology with the new Mali G2-Ultra NX, our best yet, to deliver fully ray-traced lighting and smoother gameplay with less graphics work. Coming soon to mobile. 👇 Ensure your latest flagship mobile is powered by Mali G2-Ultra NX to experience richer visuals and new levels of effciency. okt.to/irRwS7
17
the interesting part of ripwire is giving the agent callers and tests alongside the code i'd measure how often it misses an affected caller after a change fewer tokens is nice. fewer missed dependencies would make me keep it
ripwire, from Red Hat Emerging Technologies, is a remarkably substantial new approach to giving coding agents repository context without embeddings, a vector database, an LLM indexer or a daemon. The zero-dependency C++23 binary parses 21 languages with Tree-sitter and builds a deterministic structural map of a codebase, ranking symbols for the task while attaching call relationships, complexity, git churn, change amplification and test coverage. An agent can ask what matters for "incremental cache invalidation," for example, and receive the relevant symbols, their callers, likely blast radius and tests to run in a token-budgeted response instead of grepping and opening files repeatedly. GitHub Repo: github.com/redhat-et/ripwire
28
the fun bit is how loops work no bytecode saved anywhere. it jumps back and parses the source again every iteration spending cpu time to save interpreter code gets this tiny python subset into 1024 bytes of c
1
63
astra is killer for kernel perf engineering. once i told it where to look it turned the optimization in my head into a working speedup in one shot
BTW both Fable 5.1 and GPT 6 failed, overnight, on a problem that I solved in 5 minutes. I want to post what happened just to document why we're not quite on AGI yet, although getting closer every day. This post will be a bit technical because details matter So, last night Fable identified a memory leak on Bend2. A given long-running process would steadily grow in memory, until it crashed in just a few hours. That's the bug. I asked both Fable and Astra to fix it overnight. Rules: 1. Do not increase the codebase size 2. Do not cause regressions in any benchmarks Both failed. But why? First, both identified the correct root cause. (everything below is written by Fable) Bend2's runtime has 16k GPU lanes. Each lane has its own private free lists, so alloc and free are 2 stores, no atomics. That's what makes it fast. The catch: when a lane frees a block that another lane allocated, the block lands on the freeing lane's list. If that lane never allocates as much as it frees, the block sits there forever. Nothing is lost, the memory is just parked on a list nobody uses. Both models found this. Both measured it: 40-85% of all frees cross lanes. Then both spent the whole night trying to fix the allocator. Return blocks to the owner lane: +124% slower. Slabs: 73x slower. Buddy: +7-21%. Flush at sync points: +3.5k tokens. A full allocator rewrite with lock-free chains: leak gone, but +1.9k tokens and matmul 78% slower on CUDA. Every design either broke the token cap or a benchmark. Neither model converged. They just kept building more machinery on the same premise. The premise was wrong. I asked for a histogram of where the parked memory actually was. 96% of it sat in the last 4096 lanes. That's not an allocator problem. That's the scheduler: it hands out work as a pure function of lane id. Lane 0 always gets the root of the tree. The same lanes always get the leaves. The join always runs on the last child to arrive, and the same lanes always arrive last. So every launch, memory flows in the same direction, from the same lanes to the same lanes, and the flow never reverses. The allocator was fine. The lanes weren't interchangeable. Fix: rotate lane identities. Each launch, physical lane p uses the free lists of lane (p + r) mod 16k, with r advancing by a golden-ratio step. Over a cycle every identity plays every role once, and the imbalance sums to zero. No allocator change, no atomics, nothing on the hot path. A handful of lines. Once I told Fable this, it implemented it, measured it, and ran the whole gate in an afternoon: 555MB → 23MB after 2000 launches, OOM pushed from ~7k launches to ~75M, zero cost on all 16 benchmarks. So what was missing? Not capability. Both models can write a lock-free allocator overnight. What neither did was step back and ask why memory flowed in one direction at all. They accepted the asymmetry as a fact of the system and built increasingly complex machinery to undo it, instead of removing it. That's the gap: not "can it do the work", but "does it question the frame it's working in". Still, it did everything else, from the diagnosis to the census to the fix, once pointed. Closer than last year. Not there yet.
30