if you like rust and would like to build the future of low latency inference, we (the LPU team @nvidia) are hiring
nvidia.wd5.myworkdayjobs.com…
we built a ridiculously fast system in only 8 months @nvidia - and this is just the start
Artificial Analysis has measured 3,431 tokens/s on Gemma 4 31B via a private demonstration endpoint for NVIDIA Groq 3 LPX - the highest 100k-context output speed we have measured for Gemma 4 31B on either a private or public endpoint
@nvidia announced today that the Groq 3 LPX ‘interactive AI inference’ rack is in full-scale production and will enter operation later this year. NVIDIA granted us early access to a private deployment of the rack serving Gemma 4 31B for benchmarking purposes
We ran the standard 1k, 10k and 100k input sequence length prompts that we use for benchmarking serverless API providers. At both the 10k and 100k input sequence lengths, the endpoint delivered a median output speed of approx. 3,400 tokens/s, measured over 50 sequential (single-concurrency) requests. Output speed was maintained between the 10k and 100k input sequence lengths tested, indicating robust long-context inference performance
Congratulations to the @nvidia team on the launch!
The LPU team at @nvidia is building the future of low latency inference.
We're going all in on Rust. If you're a cracked Rust engineer and love making hardware do things, we want you.
nvidia.wd5.myworkdayjobs.com…
Omar Kilani retweeted
bro, you’re fine. you just need an impossible sequence of events to play out in perfect order against all odds and you’ll be fine
The Production Engineering team at @GroqInc is looking for engineers to scale and operate our global chip-to-cloud, mission-critical AI inference platform.
Own reliability from LPU silicon to global runtime: every nanosecond counts. ⚡️
Apply today: jobs.gem.com/groq/am9icG9zdD…
The Tiny TPU team (@evanliin, @XanderChin, @suryasure05 and @kennykgguo) are absolute legends.
Insanely excited to have them join the @GroqInc team. 🚀
I just wrapped up what will probably be the MOST memorable summer of my life.
tldr: built a cool project and ended up joining @GroqInc to work on distributed systems.
here's a timeline of everything that happened in the last ~6 months:
> feb 24: @evanliin explains to me (having no ml/hw background) how a systolic array works
> march - april: I learn hardware design basics and make some small projects
> may 8: @evanliin calls me and @XanderChin and says we should live together for a month and see what we cook up
> may 12: we start living together in toronto and decide to build a toy tpu that does inference and training
> may 20: we get inference fully working
> may 21 - june 12: continue chipping away at getting training working
> june 13: all of us go to sf for yc sus and meet @kennykgguo who ended up joining us
> june 19: we visit an up and coming ml asic startup and talk to some really smart engineers
> june 20 - august 4: continue working on training
> august 5: see @karpathy's challenge about building an educational resource on a unique topic so we decide to finish tinytpu and push out our article in 2 weeks
> august 10 - august 16: spend half of my time working on the tinytpu article while on an alaska cruise (top tier experience btw)
> august 17: flight back to ottawa gets cancelled by air canada strikes so I decide to go to sf
> august 18: make the tinytpu tweet before getting on my flight and noticing that it blew up after I got off my flight (also @karpathy gave us a shoutout!!!)
> august 19: get dms from @omarkilani and @JonathanRoss321 about working for @GroqInc
> august 21: have lunch with @omarkilani who basically gives us all a verbal offer
> august 22: visit the @GroqInc hardware lab and meet @GavinSherry and @jwswj
> august 27: meet @JonathanRoss321 to talk about the tpu and he gives us our offer letters in person
> sept 22: first day at @GroqInc
truly, everything that happened in the last two weeks was straight out of a movie and I am still processing it.
I cannot be more grateful to everyone we met from @GroqInc for betting on us and bringing us on.
I have a lot more thoughts on everything which I plan to write a blog post about, but my one biggest takeaway and the driving principle of everything that's happened is that you can just do things.
Great write up of some difficult root causing.
Running LLMs (especially on custom stacks, compilers, etc) and putting them on the internet turns out to be one of hard problems in computer science.
Huge respect to the teams in the trenches keeping it all running day after day.
Replying to @claudeai
In our investigation, we uncovered three separate bugs. They were partly overlapping, making diagnosis even trickier.
We've now resolved all three bugs and written a technical report on what happened, which you can find here: anthropic.com/engineering/a-…
1
11
1,135
Biggest launch in @GroqInc history.
Pre-deployed a ridiculous amount of TPS for these models, and half of it got used up instantly.
OpenAI’s open models are live and already running on Groq. Try gpt-oss-20B and gpt-oss-120B today.
Groq delivers 128K context and built-in tools such as code execution and browser search. For the first time, developers and enterprises can deploy open models backed by OpenAI instantly, anywhere, at scale.
Start building now. Links in comments.
How the gpt-oss launch went down at @GroqInc
(Also: we deployed both models across our global footprint in the US, Canada, EU and KSA)
gpt-oss is a big deal; it is a state-of-the-art open-weights reasoning model, with strong real-world performance comparable to o4-mini, that you can run locally on your own computer (or phone with the smaller size). We believe this is the best and most usable open model in the world.
We're excited to make this model, the result of billions of dollars of research, available to the world to get AI into the hands of the most people possible. We believe far more good than bad will come from it; for example, gpt-oss-120b performs about as well as o3 on challenging health issues. We have worked hard to mitigate the most serious safety issues, especially around biosecurity. gpt-oss models perform comparably to our frontier models on internal safety benchmarks.
We believe in individual empowerment. Although we believe most people will want to use a convenient service like ChatGPT, people should be able to directly control and modify their own AI when they need to, and the privacy benefits are obvious.
As part of this, we are quite hopeful that this release will enable new kinds of research and the creation of new kinds of products. We expect a meaningful uptick in the rate of innovation in our field, and for many more people to do important work than were able to before.
OpenAI’s mission is to ensure AGI that benefits all of humanity. To that end, we are excited for the world to be building on an open AI stack created in the United States, based on democratic values, available for free to all and for wide benefit.
Come work with @aarush on OpenBench:
job-boards.greenhouse.io/gro…
5
331
We’ve deployed the tool call template update to @GroqInc. Looks good. Thanks @Kimi_Moonshot 🫡
We've just fixed 2 bugs in Kimi-K2-Instruct huggingface repo. Please update the following files to apply the fix:
- tokenizer_config.json: update chat-template so that it works for multi-turn tool calls.
- tokenization_kimi.py: update encode method to enable encoding special tokens.
👉 huggingface.co/moonshotai/Ki…
❤️ Big thanks to "bchenfireworks" and "cassanof" for reporting this!
Related issues:
huggingface.co/moonshotai/Ki…
huggingface.co/moonshotai/Ki…
Getting emotional seeing this (haven’t slept in 72 hours, YOLO launched a 1T param model)
Huge credit to the incredible teams at @GroqInc for the relentless grind that made this possible.
Ridiculously excited to welcome @glenmaddern to @GroqInc 🫡
We’re going to be doing some really cool things with MCP.
Watch this space. 😎
Replying to @glenmaddern
So what's next? Well, as of last week, I am now the Head of MCP at @GroqInc! 🎉
You might not have heard of Groq—they make custom silicon for AI inference. Which makes running models on their cloud way faster than what you're probably used to. This clip is not sped up:
If this is you we’re hiring @GroqInc
groq.com/careers/
Or DM me.
How to hire Engineers who Ship Kernels!
vaibhawvipul.github.io/2025/…
Groq 💗 OSS
More to come. 🫡
Compound Beta is an awesome project led by @benklieger 🔥 To showcase the capabilities of Compound Beta I've kicked off another project called Groq Desktop Beta (fully open source!) which allows you to use local MCP servers with your favorite @GroqInc hosted models. A thread 🧵
.@benklieger has you covered for low latency agentic workloads via @GroqInc 's new compound AI system
































