Image generation on iPhone is solved.
77 images in 15 seconds on an old iPhone 13.
Our latest improvement made image generation on-device 10x faster
Only possible with QVAC.
Our engineering team just shipped an improvement that makes image generation on iPhone about 10x faster!
In the video, an iPhone 17 in airplane mode makes a Stable Diffusion 2.1-base image in 6.8 seconds, then an SDXS image in 200 milliseconds.
Everything runs on the phone.
Congrats to everyone who contributed to that. Let's make it the best engine out there for edge devices...
llama.cpp v1.0.0 is now available
huggingface.co/blog/ggml-org…
1
57
We made search in TurboVec, the vector index behind RAG in QVAC, 2x to 8.2x faster and more accurate.
We measured it with a full scan of every vector, on Apple M5 Pro, @NVIDIAAI DGX Spark and @AMDRyzen Strix Halo.
It is available now in QVAC Fabric: github.com/tetherto/qvac-fab…
gianfranco retweeted
A doctor wants to see the patient, not the keyboard. AI scribes help, but most send sensitive visit data to the cloud.
So I built a @qvac prototype: record the visit, get a draft SOAP note, all on the device and offline.
Waiting for TranscribePsy on the SDK.
The second makes Qwen3.5, 3.6 and 3.8 models read your prompt faster on NVIDIA GPUs, so the first token arrives sooner.
Their Gated DeltaNet layers now process the prompt in blocks of tokens instead of one token at a time. Qwen3.5 35B reads prompts up to 21% faster, with no measurable change in quality.
github.com/ggml-org/llama.cp…
We've been on a roll lately: two features from qvac-fabric, our fork of llama.cpp, were merged into llama.cpp this month.
The first is for PCs whose dedicated GPU is too small for a big mixture-of-experts model. The model's experts stay in system RAM, and the GPU keeps a cache of the ones it used most recently.
On an RTX 5090, Qwen3.8-Flash-Next (about 100 GB, three times the card's memory) now generates text 2.2 times faster. To try it, pass --moe-cache-mib with a size, next to -cmoe or --fit.
Thanks @ggerganov for the merge.
github.com/ggml-org/llama.cp…
Every sound in this video except the narration was made locally on a laptop: roosters crowing at dawn, two people talking, an excited voice, and a dog barking under a conversation.
They come from three open models by @Open_MOSS and @MosiAI_Official, running locally with QVAC.
Thanks to @xpqiu and the team for open-sourcing them.
What each model does ↓
We optimized NVIDIA's Parakeet 0.6B models to transcribe twice as fast on the @Snapdragon 8 Elite, a flagship @Qualcomm phone chip.
We moved the work from the chip's graphics processor to its AI processor, the Hexagon NPU. On a OnePlus 13s, Parakeet CTC 0.6B now transcribes a minute of audio in 970 ms, with the same text the GPU produced.
Thanks to @NVIDIAAI for Parakeet and @ggml_org for the code we built on.
github.com/tetherto/qvac-fab…
Another PR of ours just merged into llama.cpp: the fast Metal kernels from Monday now cover all 26 weight formats the Mac GPU runs.
They made speculative decoding faster on Mac for 10 formats, and this PR adds the other 16, including BF16, MXFP4, Q2_K, Q3_K and the IQ quants.
On an M3 Ultra, matrix multiplies run up to 4.4x faster when a model checks several tokens at once.
Thanks @ggerganov for asking for it and merging it within a day.
github.com/ggml-org/llama.cp…
You can now run one AI model across the GPUs of your devices, whatever operating system they run: macOS, Linux, Windows, Android or iOS.
In the video, Qwen3.6-35B-A3B, a 21 GB model, runs across a Samsung S25, an iPhone 16e and a MacBook Pro. None of them could load it alone (8 GB of memory on the iPhone, 12 GB on the S25, 16 GB on the MacBook), so each device takes a slice of the model, and together they answer four prompts at once.
A Linux laptop and a Windows PC can pool their memory the same way. It's in the QVAC SDK 0.21, free and open source.
Built on llama.cpp's RPC backend, thanks to @ggerganov and @rgerganov. @digitalix, want to try it on your Macs?
How it works ↓
We have added Cluster inference support to Qvac Fabric. You can cluster together multiple machines and split the inference across them. Supporting both tensor and layer parallelism. Here 2x DGX Spark running Deepseek-v4-flash-0731 over RDMA.
github.com/tetherto/qvac-fab…
Latest @qvac Fabric work added CoreML support to make Parakeet insanely fast. You can transcribe 25 mins audio in nearly 16 seconds on an Iphone 16.
check github.com/tetherto/qvac-fab…
QVAC Fabric now runs Parakeet's encoder through Core ML on Apple silicon, which moves it from the GPU to the Neural Engine.
On a MacBook Air M2, Parakeet TDT 0.6B v3 went from 78x real time on the Metal GPU to 164x through Core ML.
1
139
Parakeet Redux is compressed from NVIDIA's Parakeet TDT 0.6B v3, so we ran it against the original, which QVAC runs in Q8.
Both transcribed the same two hours of audio on the GPU of a MacBook Pro M5 Max.
QVAC took 22.3 seconds, 323x real time, and Redux took 44.2 seconds, 163x. Both times include loading the model.
QVAC SDK 0.20 is live. 🚀
We added support for:
- ABot-World: generates a walkable world from a single photo
- MiniMax-H3: create video with synchronised soundtrack
- Video models can now run on GPUs with less VRAM
- TranslatePsy-AfriSLM: translate between English & 19 African languages on-device
- ACE-Step can now generate entire songs from a single sentence & more new features
We also merged the latest changes in llama.cpp into QVAC Fabric, to ensure continuous compatibility.
Learn more in the thread below.
Complete release doc here: github.com/tetherto/qvac/rel…
The QVAC audio stack is continuously optimised. We compared its performance against several other open-source solutions, and QVAC comes out fastest on transcription, speech synthesis and music generation, across every GPU lane we measured.
If you want to build fast apps with local audio AI that fit on consumer devices, you should definitely check QVAC out.
github.com/tetherto/qvac-fab…
Did you check latest inference optimizations on @qvac for speech models ?
48
Deepseek v4 flash is now available on Fabric. We added Vulkan and Metal support, so you can run it on any gpu. Performance improvements come from optimized fused kernels. Go and try the new @deepseek_ai released version '0731' on github.com/tetherto/qvac-fab…











