🧑💻 Voice AI @trydaily // ᓚᘏᗢ @pipecat_ai 🎹 I have too many synths
London, UK
Joined July 2008
- Tweets1.5K
- Following609
- Followers1.6K
- Likes543
New week, new tiny model for Pipecat.
Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm.
It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50 ms per turn. In a Pipecat pipeline it runs beside the text tone reader, which reads what was said in the context of the previous exchange, providing broader context to the harness and metering.
This is one area where a cascaded pipeline has a real edge over speech-to-speech... each turn gets explicit, calibrated readings of both the words and the delivery, with context of the single turn and the entire conversation.
Available in two weights (base and tiny): huggingface.co/pipecat-ai/vo…
Phonon from @GradiumAI is an open TTS model in 31M and 40M sizes, perfect for on-device! It’s voice promptable too, via Gradium’s Voice Designer.
I love experimenting with voice and mobile. Here is Little Voices, an app where you can ask the angel and devil on your shoulder anything. It's a @pipecat_ai app in embedded Python on iPhone.
• Apple SpeechAnalyzer for speech-to-text
• Apple Foundation Models for the LLM
• Gradium Phonon 31M for TTS.
• Pipecat running in Python on the iPhone
• Pipecat Audience (for who speaks next. It works out whether you’re talking to the angel or the devil.)
• No network or inference costs!
📖 Repo here: github.com/pipecat-ai/ios-li…
A few more thoughts on small models for intent, movement and tone pipecat.ai/labs/intent-and-t…
Using purpose-trained classifiers that are small and fast enough to not impact the voice loop can help close the gap in naturalness between a speech-to-speech and cascade pipeline.
For intent and tone though, text obviously only gets you so far. But audio models can be small too!
Check out Pipecat Backchannel Detector: huggingface.co/pipecat-ai/ba…
It's just 734k parameters, and does a pretty great job of figuring out when a user "mhmm"s or "oh right"s without interrupting the bot.
Oh, and these backchannel events are all part of the RCS and RTVI standard we have defined in Pipecat, so supported with typed events in the client libraries too!
We released Reader & Tagger, two tiny CPU-based classifier models that allow you to implement @pipecat_ai's Realtime Context Standard with any LLM (especially smaller ones, which are bad at following providing structured output without fine-tuning.)
This demo shows parsing the tone of the user transcript, metering their mood, and steering the LLM and TTS (with style tags.)
We also use it to include sounds such as chuckles or breaths, where they would fit naturally in the conversation.
Building on @vr000m 's amazing pipecat-local-tts-server, here is a Pipecat MacOS menubar tool for downloading and running local models for use in your agents.
I'm the sort of person that runs Postgres through something like postgres.app, so I needed this.
Please let me know if you'd find this useful so we can justify the time getting the app certified!
(and yes, it was just an excuse to get the catmark in my toolbar)
New Pipecat UI component: duplex timeline. I've been using this in dev for a while to visualize turns, latency, interruptions and back channeling.
ui.pipecat.ai/docs/component…
Our shadcn registry is "pipecat", if you're a CLI person.
npx shadcn@latest registry add @/pipecat
Pipecat PhoneLLM Alpha 2 🎉 now in Mini and Max sizes!
Max is a "microthinking" model: one sentence of reasoning keeps multi-step conversations accurate yet still under 200ms TTFAT.
More about it here:
pipecat.ai/labs/phonellm-alp…
Weights here:
huggingface.co/pipecat-ai
"Oh great, another release." New in Tone RC4: sarcasm detection 💀
Confidence output matters a lot here, as not all sarcasm is created equal, and when it calls sarcasm, it’s right 86% of the time (... or in the spirit of this tweet, 86% of the time, it's right every time!)
RC4 also comes as 16-bit weights now too at just 56 MB (half the download) with the same tags and the same speed on CPU.
Weights on HF: huggingface.co/spellspeak/to…
Demo: github.com/spellspeak/models…
Here’s Tone, another tiny decision model, fine-tuned from MiniLM and running in a Pipecat pipeline. It reads every line of a transcript for emotion, intent and hostility, and the harness uses that to track how a character feels about you as you talk. ~10ms on CPU.
In this demo, I'm using the output to pass tone guidance to a TTS service.
Confidence meters matter a lot for this model as it needs more training on sarcasm and subtle digs. For example, this character often responds with anger when I greet her with "Hello Nova, you're looking very orange today."
Model here: huggingface.co/spellspeak/to…
I’m obsessed with using tiny classifier models in Pipecat agents. This one, Audience, is a ~33M-param ONNX model built on Ettin that works out who each line in a scene is for. It uses names, descriptions, nearby items and where people are standing, and it can pick one character, several, the whole group, or return "unclear" if you're like me and talk jibberish.
It runs in ~6–7 ms on CPU and slots nicely into a Pipecat pipeline. Multi-character agents are really fun!
Weights here: huggingface.co/spellspeak/au…
Pipecat CXX Client SDKs and transports! Spatial audio and world data from Unreal <> Pipecat are too much fun to experiment with.
My gamer instincts took control for a demo, but the scope is much wider (robotics, embedded, etc.) Building stuff like this is just Pipecat, with the same events and pipeline code we're all used to. Sidecar it, bundle it with local weights, so many ideas.
Oh, and Opus made the demos. Mad how good it is at creating beautiful scenes and interactions in Unreal with very little prompting.
🔗 C++ client SDK: github.com/pipecat-ai/pipeca…
🔗 Unreal demo: github.com/pipecat-ai/pipeca…
🔗 Pipecat Room (multi-character demo): github.com/jptaylor/pipecat-…
Jon Taylor retweeted
This is awesome. Multi-character interactions in Unreal Engine, completely LLM-driven and unscripted. You can spin your own version of this up pretty easily.
This is another Pipecat + Jev experiment from @JonPTaylor.
Watch Jon walk around the game world, talk to the characters, and ask them to do things. They interact with Jon, with each other, and with the world.
Tech stack:
- One Pipecat agent process syncing state with Unreal via the Pipecat RTVI protocol.
- Three non-player characters. Each character is a Pipecat worker with its own LLM prompt and tools. Each character also has its own TTS service so its voice is unique.
- When Jon (the human) talks, Jev assesses Jon's intent and which character(s) he is talking to.
- Unreal 5.8 with Spatial Audio. (Watch the video with headphones on!)
- Pipecat C++ client embedded in Unreal Engine.
Little Pipecat voice companion to talk to whilst playing WoW Forever. It responds to game events, gives me suggestions, tells me lore, reads out quest text (in dramatic fashion)... and of course has different avatars and personalities.
It's a little janky, but definitely fun to have something to talk to when playing solo. It fetches Lore from various web sources, so some responses are bit too slow as you can see at the start.
I really didn't want to break any of Blizzard's EULA here, and using real screen vision is forbidden (to prevent botting, I guess?) so I wired up a hotkey that snaps a screenshot that it can look at to sort of fake it.
Unsure about sharing the code, incase it encourages the wrong sort of behavior, but maybe someone can tell me if this is is T&Cs friendly?
Weeks on and I'm still finding awesome ways to use @typesafeai Jev in Pipecat. This demo is ONE bot pipeline, with a classifier front-running the LLM to determine who the user is talking to. Pipecat then orchestrates a handover between two 'character' workers.
Oh, and of course, Opus 5.5 one-shotted this idea. Code might be ugly: github.com/jptaylor/pipecat-…