@fluidinference , AI Researcher https://nitter.cf/t.co/8ko2FKWp4A
Joined July 2025
- Tweets431
- Following310
- Followers1.2K
- Likes345
Vela 3.0, now on-device on Apple silicon π‘
vela3-fast (0.4B) runs on Core ML: you name the options or the spans, and it answers in one encoder pass on the Mac's GPU.
β‘ ~10 ms per decision, ~18 ms per PII pass over a paragraph (M5 Pro)
π― 1,200 / 1,200 decisions idenitical to the base, with identical PII spans
π Redacted a 20,000-word case file at ~0.5 ms a word: 2,956 / 2,971 planted names, cards, SSNs and addresses caught, nothing leaves the machine
π€ Weights: huggingface.co/FluidInferencβ¦
𧩠Swift : github.com/FluidInference/Flβ¦
Introducing Vela 3.0 π‘ Name it. Vela finds it.
Six open models (0.3Bβ27B): decisions + the spans you name. One call.
β‘ 380M: #1 Decision Index β€500M
π― Beats GPT-5-mini at zero-shot classification
π 18 ms per span question
π€ huggingface.co/collections/vβ¦
#Jev #Decison
3rd merged PR in NVIDIA Dynamo π
every DCGM scrape was leaking a socket. Now the operator reuses one HTTP client instead of a fresh transport per scrape.
github.com/ai-dynamo/dynamo/β¦
LFM3-1.7B just read the whole of Pride and Prejudice on a M5 Pro using MLX.
It read 100k tokens in about 28 s (sped up 8Γ in the clip), then answered 100 questions back to back in 27 s, including finding a password hidden somewhere in the book.
huggingface.co/alexwengg/LFMβ¦
Introducing π»π΅πΌπΉ: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.
> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> π»π΅πΌπΉ-π·.π½π±: text, 256k context window.
> π»π΅πΌπΉ-π
π»-πΈπ±: vision-language model with video-input support for spatial understanding
> π»π΅πΌπΉ-π°ππππ-πΈπ±: tool calls, voice chat, reasoning in 18 languages.
> π»π΅πΌπΉ-πΎπππ-πΈπ±: an omni model combines vision, audio, and language.
> ππ·-πΈπ± and ππ·-πΎπππ-πΈπ±: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> π»π΅πΌπΉ-π
π»-πΈπ± is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> π»π΅πΌπΉ-π·.π½π± is #1 on the Pareto frontier of device intelligence πππππππ.ππππππ.ππ
Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.
1/ π§΅
LFM3-1.7B-Base writing non-stop for 60 seconds on a M5 Pro.
~9,800 toks/min, ~165 tok/s sustained, 1.3 GB of memory. MLX, 5-bit.
huggingface.co/alexwengg/LFMβ¦
Introducing π»π΅πΌπΉ: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.
> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> π»π΅πΌπΉ-π·.π½π±: text, 256k context window.
> π»π΅πΌπΉ-π
π»-πΈπ±: vision-language model with video-input support for spatial understanding
> π»π΅πΌπΉ-π°ππππ-πΈπ±: tool calls, voice chat, reasoning in 18 languages.
> π»π΅πΌπΉ-πΎπππ-πΈπ±: an omni model combines vision, audio, and language.
> ππ·-πΈπ± and ππ·-πΎπππ-πΈπ±: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> π»π΅πΌπΉ-π
π»-πΈπ± is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> π»π΅πΌπΉ-π·.π½π± is #1 on the Pareto frontier of device intelligence πππππππ.ππππππ.ππ
Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.
1/ π§΅
NVIDIA Nemotron 3.5 Transcribe, ported to Apple's Core ML and Core AI on M5 Pro zero day launch.
Core ML (video): a 49.5-min, 3-person meeting β speaker-labeled transcript with word timestamps in 6.0s , 500 rtfx .
- Speech encoder + CTC head on the GPU
- Nemotron-3-Diarization on the Neural Engine, in parallel
- ~1.5 GB, no LLM
Core AI: same encoder + CTC head as a 1.3 GB .aimodel. Core AI, Core ML and PyTorch give the same transcripts on LibriSpeech (4.62 / 4.64 / 4.64% WER, 400-utt subset). 234x vs 238x on GPU.
Audio: AMI EN2002c (CC BY 4.0).
Meet NVIDIA Nemotron 3.5 Transcribe ποΈ
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyoneβs talking at once.
1/ π§΅
Alex retweeted
Native Core ML & Apple Neural Engine Acceleration
This Core ML conversion brings EmbeddingGemma 2 natively to macOS and iOS for maximum efficiency.
Using fixed-shape functions, the text encoder runs 100% on the Apple Neural Engine (ANE) without CPU fallback, while the audio and vision encoders run hardware-accelerated on the GPU, complete with Matryoshka dimension truncation (768 down to 128).
Built by @Alex_tra_memory (Fluid Inference): nitter.cf/Alex_tra_memory/statusβ¦
Hugging Face: huggingface.co/FluidInferencβ¦
We got EmbeddingGemma 2 running 100% on the Apple Neural Engine π M5 Pro
10,080 posts sorted into topics live, ~620/s on a MacBook, then any topic splits into subtopics in under a second. Fully on-device.
π€ huggingface.co/FluidInferencβ¦
π» github.com/FluidInference/Flβ¦
got my 2nd Nvidia PR merged in π₯³ . Fixed a memory bug in Dynamo's multimodal embedding cache with Qwen3-VL, a 1 GiB cap could hold ~4 GiB of host RAM.
DeepStack rows are ~4Γ wider than the hidden size the cache charged per row. It now uses the width vLLM measures.
github.com/ai-dynamo/dynamo/β¦
Gave d3-ultra a cluttered utility website and four errands: pay the bill without fees, report an outage, find the gas line, switch to paper bills.
16 clicks, no wrong turns, ~0.5 s per decision, 0 tokens generated. Every click is a probability.
Meet d3-ultra: our newest open 27B multimodal decision model.
#2 of 230 on the Jev Decision Index text board, +3.03 over d3.
Text, images, video: it boxes what it finds and tracks it through time.
Open weights (Apache-2.0): huggingface.co/vllm-sr/d3-ulβ¦
#Jev #Decision
currently using claude to do verbal interviewing while asking deepseek for questions to test around. honestly deepseek sure shows you the endless depth of inference engines and model concepts
been exploring around and see how different inference engines work.
Qwen-Image-2.1 draws the character and drops it into a real clip's first frame. Wan2.2-Animate-2 (distilled, 10 steps) copies the athlete's flips frame for frame. One RTX A6000 + SGLang.
i will eventually store all these coding demos on a repo.
Looking back it was quite the emotional high To gain this many followers but the marginal between the first 1000 and the next 1000 is significantly diminished in reach and emotional returns.
The next possible notable count would be 10,000 followers but the goal should still be to network max . Quality over quantity or amount of interesting to network with
Nanbeige4.3-33B Mixed 3/4/6-bit MLX: 15.7 GiB, runs on a 24 GB M5 Pro π
Here it is as a terminal agent ~25 tok/s, fixing a buggy repo end to end β¬οΈ
huggingface.co/alexwengg/Nanβ¦
πExcited to announce Nanbeige4.3-33B!
To push the performance ceiling of small agents, our strategy in Nanbeige4.3 starts with building a powerful teacher first.
Nanbeige4.3-33B delivers SOTA agentic performance among open models at ~30B parameters:
* Strong results on agentic coding and cowork benchmarks.
* When integrated into mainstream agent harnesses, it tackles real-world tasks with cost-efficient inference.
* It will serve as a teacher for upcoming smaller Nanbeige models, starting with Nanbeige4.3-3B.
huggingface.co/Nanbeige/Nanbβ¦
huggingface.co/Nanbeige/Nanbβ¦
#LLM #Agent
Nanbeige4.3-33B as a live coding agent on an A6000:
Given a Dijkstra function with 5 planted bugs + 4 tools, it fixed all 5 and passed 8/8 hidden edge-case tests it never saw. The map replays its code after every edit.
vLLM 0.32 Β· FP8 weights Β· NVFP4 KV cache on Ampere (113K-token pool) Β· ~19 tok/s
πExcited to announce Nanbeige4.3-33B!
To push the performance ceiling of small agents, our strategy in Nanbeige4.3 starts with building a powerful teacher first.
Nanbeige4.3-33B delivers SOTA agentic performance among open models at ~30B parameters:
* Strong results on agentic coding and cowork benchmarks.
* When integrated into mainstream agent harnesses, it tackles real-world tasks with cost-efficient inference.
* It will serve as a teacher for upcoming smaller Nanbeige models, starting with Nanbeige4.3-3B.
huggingface.co/Nanbeige/Nanbβ¦
huggingface.co/Nanbeige/Nanbβ¦
#LLM #Agent
study report number 3 about model inference engines.
- weights load from disk once and stay on the GPU & anything to do with the tokens on the gpu too
- the CPU orchestrates using the scheduler for example which is on the RAM
- a model judging itself is biased, and a rubric and reference helps alot
- quantization still risk regressions thats why you gotta check the probability distributions, token outputs by greedy decoding,
- Query not cached: itβs read once, while keys and values are reread by every later token.
- MoE big model on GPU but with the speed of a tiny model
- RoPE is baked into training, past its training token limit it can produce corrupted outputs.
- how do you calculate KV cache size? using vllm & sglang for example. its (gpu size) * 0.90 - model size - 2 gb( for cuda & local variables) = kv cache size .
next steps:
- better get why Query is so different from KVs, still don't the exact differences
- more details about what part of the model lives on the gpu and ram .
- conditions where the model is slow or becomes dumber all the sudden
- what the schedule exactly does
- more details on RoPe
I am currently trying this new study plan when it comes to learning code. I ask ai to generate a simply buggy script and give me a list of bugs to find and fix.
I just donβt have the drive to write code but debugging feels more motivating . I think my learning style is to prefer almost instant feedback .
Merged my first ever fix into Emmy, @CloudRiftAI's open-source PyTorch-to-CUDA compiler.
When fusion dropped a store, the program still compiled and silently returned unwritten memory. Now it fails loudly and names the kernel.
Thanks @ditrifonov for the review!
github.com/cloudrift-ai/emmyβ¦
alright study time over for today:
- switching between different models on the inference engine is a real consideration
- while harness handles deadlocking, resource contention can still happen especially if you have TTS, ASR and LLM on the same GPU
- alot of challenges revolve around the GPU having the resources
- in ASR to LM to TTS. the asr always gets first priority in proceeding to ensure smooth user exp.
- context length or kv cache can be set by engine
- kv cache Key and Value numbers are stuff only machines can grasp not us humans.
alot of it is identifying bottleneck for resources and speed and random issues popping up like the LLM part freezing up or LLM suddenly getting dumber.
being able to identify supporting mlx vs lama.cpp.
currently my mental model of the inference engine is this .
ASR β scheduler β LLM with its KV cache and cache manager β TTS
tmw is probably use vLLM and SGLang repos as point of reference when designing inference engines .
- understanding why query isn't stored but kv cache are
- what the scheduler does
- MoE
- RoPe
- cases when model gets dumber
trying to understand how inference engines lately. yesterday i was clueless as a headless chicken now i have a visual of it .
- scheduler + LLM model + kv cache/managers .
- learning models can do parallelism shit weights are just being read & models can do batch decoding outputs too.
next steps are to the many issues inference engineers ecnounter, have like kv cache getting filled up.
honestly its like grasping how computers work or ram management all over again .