Frontier AI on local hardware. EXO 1.0 is now open-source (Apache 2.0): https://nitter.cf/t.co/SGGGK784Qp
Joined March 2024
- Tweets710
- Following3
- Followers55K
- Likes3.5K
Pinned Tweet
The future of AI is open source and decentralized
"Exo's use of Llama 405B and consumer-grade devices to run inference at scale on the edge shows that the future of AI is open source and decentralized." - @mo_baioumy nitter.cf/ac_crypto/status/18159…
EXO Labs retweeted
In June @NaderLikeLadder emailed Jensen about a crazy idea we had over dinner to make Local AI 10x better on NVIDIA hardware.
Jensen replied in 7 minutes. He saw this as important work, and approved a war room for us to work on this at NVIDIA HQ.
For a month, NVIDIA HQ was the @exolabs satellite office. We were overwhelmed by the support we got. We were back to back the first week with smart NVIDIANs coming in to our war room to help us. Creators of CUDA, Dynamo, people who built NVIDIA.
Long story short with NVIDIA we managed to improve their Local AI playbooks by 9.6x (almost the 10x goal we set out with). We’ll be publishing a whitepaper on this work.
We were just a small startup with a crazy idea. Jensen took a bet on us. Jensen cares deeply about Open Source AI and Local AI.
I don't believe Jensen is replaceable, he's a 1 of 1 human, but I hope NVIDIA will continue to aggressively support Open Source AI and Local AI, and continue to take bets on small startups with crazy ideas.
Thank you Jensen
EXO Labs retweeted
GLM-5.3-Flash in a backpack
Okay last one for today.
running on 2x M5 Mac Ultras (256gb each)
- single stream 150 tok/s
- 8 concurrent streams 380 tok/s
- 1.4 million token kV cache
This has been powered by 2x e-bike batteries for over 90 minutes generating nonstop
EXO Labs retweeted
Sam Altman’s AI predictions rely on a vision for “recursive self improvement” that does not yet exist, at least not publicly.
On TWiAI 36, the panel considers whether or not one frontier lab really could run away with the entire AI game, or if Altman’s “3 pathways forward for AI” prediction rests on a faulty premise?
Are the major labs talking about regulation so much because they’ve seen what kinds of models are coming next? Or just because regulation is good for whoever gets out to an early lead?
Expert panelists Jaleh Rezaei (Mutiny) and Alex Cheema (EXO Labs) join Jason and Lon for a brand-new This Week in AI.
@jason, @jalehr, @alexocheema, @lons, @mutinycorp, @exolabs
0:00 Cold Open / Show Intro
0:58 PayPal - Pay zero processing fees on your first $100K in eligible PayPal payment volume. Learn more at paypal.launch.co
2:56 How much hardware it takes to run local AI models
4:32 DHS orders DeepSeek out of federal systems
5:53 Why enterprise clients are uneasy about Chinese models
9:47 Could an open-source model "phone home" to Beijing?
14:10 Why enterprises are much more price sensitive about AI tools
16:39 Are Sam Altman's AI endgame theories "deranged"?
25:45 Why the major labs want more regulations
26:28 The ElevenLabs Archer controversy continues
34:30 Who actually owns the rights to your voice?
45:17 "Substack for AI Models"
46:20 China needs more chips than Nvidia can make
52:40 Are enterprises running more local models?
🎥 Watch the full episode here 👇
EXO Labs retweeted
The memory unit economics of Apple Silicon are insane.
When you realise LLM inference is massively memory-bound (even at high concurrency!), you find that $ per memory bandwidth ($/GB/s) is the most important metric for inference.
M5 Ultra: $5499, 96GB @ 1.2TB/s, $4.5/GB/s
DGX Spark: $6950, 128GB @ 273GB/s, $25.5/GB/s
DGX Station: ~$150K, 252GB @ 7.1TB/s (HBM3e), $21.1/GB/s
M5 Ultra has 5.6x better $/GB/s than DGX Spark and 4.7x better $/GB/s than DGX Station.
I am surprised Apple didn't increase their prices more (maybe they will so might be a good time to buy...)
I expect 512GB M5 Ultra Mac Studio to start at $15899 (following the usual +$25/GB incremental memory price). That's a crazy good price for that much high speed memory.
With RDMA over Thunderbolt 5 you get single digit microsecond latency and you can connect up to 5 Thunderbolt 5 cables between Macs for 400Gb/s network bandwidth (same as ConnectX-7). Thunderbolt 5 cables are ~$25 each.
The main problem is the software on Apple Silicon is not great. There are about 30 competing inference engines, each supporting a small set of models, and quite unstable. The ecosystem around MLX is immature and limited (e.g. no Dynamo, LMCache, llm-d, ...). In our experience, more serious customers like Governments and Enterprises are put off by that and want something stable with an ecosystem of tools and software to run inference in production.
My mind is blown by how far ahead Apple is with local LLMs. The new Mac Studio M5 Ultra will have a 512 GB model for around 15-20k. You can chain 6 of them together using thunderbolt 5 and this have over 3 TB of ram and a massive amount of CPU and GPU cores. This means you can run a frontier model with over 2 trillion parameters locally for around 100kish. Nivida is around 500k for the same capability.
EXO Labs retweeted
Really great little DGX Spark guide from @0xSero
read it on HF huggingface.co/blog/exolabs/…
💛
Just @alexocheema and @exolabs rocking it, and with yet more coming down the pike
reuters.com/business/retail-…
EXO Labs retweeted
I interviewed DHH, the creator of Omarchy - The Agent Operating system + Ruby on Rails, one of the most important frameworks of our time.
He's a father, builder, race-car driver, and such an inspiring person.
One of the most positive people I've had the honor of meeting.
EXO Labs retweeted
My GTM lead asked me to share this. For serious business people pls.
tally.so/r/lbx4l6
EXO Labs retweeted
@johnternus has the clear path to owning sovereign AI on the desktop and phone thanks to Jobs making uber own silicon AND ramping up support for @exolabs AND putting 128gigs+ in machines
Predict they buy @perplexity_ai
EXO Labs retweeted
Apple's AI strategy is Apple Silicon.
The iPhone, iPad, MacBook, Mac Mini and Mac Studio all use the same hardware architecture.
Apple Silicon is energy efficient, quiet, and the memory unit economics are incredible.
Apple has leaned hard into Local AI, now serving every segment of Local AI, from SLMs running on an iPhone/iPad to 2T models running on a cluster of 4 x 512GB M5 Ultra with RDMA using @exolabs (the cluster has 2TB memory @ 4.8TB/s).
Most inference will run locally, and Apple wants it to run on their silicon.
The main issue I see right now is the software. We need a stable, canonical inference engine rather than 10 unstable, incomplete ones. Who is solving this?
@johnternus has the clear path to owning sovereign AI on the desktop and phone thanks to Jobs making uber own silicon AND ramping up support for @exolabs AND putting 128gigs+ in machines
Predict they buy @perplexity_ai
EXO Labs retweeted
hiring a founding designer for @exolabs
in-person in SF
$20k referral fee
jobs.ashbyhq.com/exo/2d9292b…
EXO Labs retweeted
Interesting article in @theinformation.
Macs are Apple's fastest growing business right now, driven by massive AI demand.
Apple's AI strategy IS the Mac.
Trillion dollar opportunity imho.
theinformation.com/articles/…
Before the patch, Linux detected the Mac but used the generic CDC-NCM config. This expected an interrupt endpoint but the Mac doesn't provide that, so the driver failed to attach and no network device would appear.
Previously there were hacky workarounds like manually patching the kernel.
This small patch makes it so Linux exposes usable network interfaces out of the box.
github.com/torvalds/linux/co…
Our patch enabling direct USB-C networking between Apple Silicon Macs and DGX Spark has landed in the mainline Linux Kernel.
This simplifies heterogeneous Local AI setups like P/D disaggregation over an ordinary USB-C cable.
github.com/torvalds/linux/co…
EXO Labs retweeted
why do I have four mac studios?
because my inference cost is zero dollars.
> 291B parameters
> tensor-sharded across four ultra chips
> wired in a thunderbolt 5 ring
> at 80Gbps with RDMA
sorry anthropic
EXO Labs retweeted
Someone posted this on reddit and now it’s #1 on r/LocalLLM!
Answering questions about exo and local AI in the thread.
exo featured on Apple's new M5 Ultra Mac Studio and M6 / M5 Pro Mac Mini pages.
Over the past year, we have worked closely with Apple on low-latency RDMA networking over Thunderbolt 5, enabling clusters of Macs to run massive models like Kimi K3 and GLM-5.3 at API speeds.
With RDMA, aggregate memory bandwidth across Macs scales ~linearly.
A cluster of 4 x M5 Ultra Mac Studios scale to an aggregate memory bandwidth of ~4.8TB/s. Previously, these were speeds only achievable with data center GPUs.
Our vision is a data center on every desk. Apple Silicon's superior memory unit economics, power efficiency, and out of the box experience for Local AI make that possible.
Thank you to @angeloskath, @awnihannun, @twid and countless others at Apple who tirelessly to bring this technology to the world.
EXO Labs retweeted
exo is currently top of r/LocalLLM (in response to this tweet).
answering questions there in detail on my alt: Longjumping_Crow_597.
feel free to ask any questions, exo or local AI related. link: reddit.com/r/LocalLLM/commen…
exo featured on Apple's new M5 Ultra Mac Studio and M6 / M5 Pro Mac Mini pages.
Over the past year, we have worked closely with Apple on low-latency RDMA networking over Thunderbolt 5, enabling clusters of Macs to run massive models like Kimi K3 and GLM-5.3 at API speeds.
With RDMA, aggregate memory bandwidth across Macs scales ~linearly.
A cluster of 4 x M5 Ultra Mac Studios scale to an aggregate memory bandwidth of ~4.8TB/s. Previously, these were speeds only achievable with data center GPUs.
Our vision is a data center on every desk. Apple Silicon's superior memory unit economics, power efficiency, and out of the box experience for Local AI make that possible.
Thank you to @angeloskath, @awnihannun, @twid and countless others at Apple who tirelessly to bring this technology to the world.