The next wave of AI is real-time—voice, video & the physical world. Follow the builders shaping it, what they’re building & how it works. Presented by @agoraio.
Joined October 2024
- Tweets253
- Following342
- Followers208
- Likes214
What if your daily briefing became the starting point for a personal voice agent?
.@OriloEchotime is building a voice-first AI companion that combines personalized information, conversation, and everyday actions.
Start with stories that matter to you. Ask questions, change the direction of the conversation, or turn something you hear into a reminder or calendar event.
It's an interesting approach to personal AI: give people a reason to talk to their agent every morning, then make those conversations useful beyond the briefing.
#BackchannelSignals
Introducing Orilo. Your Voice-native personal agent.
No endless scrolling. No typing prompts. Just talk naturally, all hands-free. Orilo knows you, thinks ahead, and gets things done.
☀️ Start your day with a briefing made just for you.
⚡ Stay ahead of what matters. Let Orilo track it for you.
✉️ From conversations to actions — emails, calendars, and more.
Orilo is now live on the App Store.
📲 Download Orilo: orilo.onelink.me/Umxq/video
NVIDIA's Rick Grandy gave his garage door a voice for Halloween—and a weakness for "please."
Meet MonsterHouse, a grumpy pirate-house projection that talks to trick-or-treaters. Say "please" and it softens. Say "thank you" and it sends you off warmly.
Making the house stay in character takes more than a prompt.
• Personality — Nemotron 3.5 Bolt E4B with a LoRA trained on thousands of example conversations.
• Memory — application logic tracks each encounter, so visitors don't have to win the house over twice.
• Voice — Magpie TTS Zeroshot gives the monster its voice, while Audio2Face-3D animates its mouth in Unreal Engine 5.
One neat trick: clean speech drives lip sync before monstrous effects are added to the audible voice.
Rick and @Merry_Yam shared the build and its engineering details:
huggingface.co/blog/nvidia/n…
#BackchannelSignals
It's a big day for decision models in real-time AI.
Tazo, Hume, and Levanto all announced new decision models on November 2.
Not every AI decision needs a generated response. These projects explore a different approach: models that make specific judgments rather than generate replies.
For voice agents, that could mean deciding when to speak, detecting frustration, or determining when a human should step in.
Three different approaches:
• Tazo — real-time decisions from live audio.
• Hume — judgments about speech, sound, and human perception.
• Levanto — decisions that can be updated as new text arrives.
Three approaches to decision models, below.
#BackchannelSignals
.@hume_ai introduced an Audio Decision API that asks questions directly about speech and sound, without relying on transcripts.
Does a caller sound frustrated? Are speakers talking over each other? Where does someone laugh?
Developers define the questions and possible answers. The model returns probabilities, ratings, or timestamps for detected events.
Its distinctive idea is human perception: for subjective questions, the probabilities estimate how human listeners would judge the audio.
The API currently processes audio files, including clips captured from live calls. A public playground is available, with API access by request.
hume.ai/blog/introducing-hum…
.@levantolabs is bringing decision models to streaming text with Sage2.
Instead of generating a response, Sage2 evaluates a question as new context arrives.
Imagine a support conversation:
"One person can't sign in." No escalation needed.
"The entire EU workspace is affected." Now the decision changes.
Levanto says Sage2 retains session context and reuses cached computation as new text is appended.
For voice agents, this could allow a separate model to keep evaluating incoming transcripts while the conversational LLM handles the reply.
levanto.ai/model
A voice agent needs to know more than whether someone is speaking. It needs to anticipate who will speak next.
@GabrielSkantze has released Mimi VAP, an open-source turn-taking model that continuously predicts the future speech activity of both participants in a conversation.
Building on Voice Activity Projection (VAP), introduced with Erik Ekstedt in 2022, the new model replaces the original CPC encoder with Kyutai's frozen Mimi audio encoder, also used in Moshi.
Every 80 ms, it estimates each speaker's current activity and relative future activity up to 2.4 seconds ahead.
These predictions can help distinguish a pause within someone's turn from an opportunity to respond or give a backchannel.
A few details:
• 18.3M trainable parameters on top of frozen Mimi.
• Trained on roughly 6,630 hours of two-speaker conversations.
• Learns from conversational speech activity without transcripts or explicit turn-taking labels.
• Supports streaming inference, with MIT-licensed code and downloadable weights.
github.com/gabriel-skantze/m…
#BackchannelSignals
Why do we keep following Voice AI hardware alongside research into full-duplex speech models?
Hardware evolves at a different pace, making each shift easier to notice: from passively recording what we say to using Voice AI as an interface to get things done.
The focus now is on making voice-to-action more reliable and easier to use, even in devices designed not to talk back.
They may not offer the full-duplex experience we're hoping for, but each step toward more useful interaction is worth following.
#BackchannelHardware
An idea, a client message, a team follow-up.
Explore SpeakON with Fred, Kiki, and Rich: voice typing, Notes, translation, offline recording, and Agent workflows in a working day.
What would you try first?
speakon.app/x
What if losing your voice didn't mean losing the ability to have a natural phone conversation?
At Cal Hacks 13.0, Thai Tran, Austin Wang, Frank Guo (@gottaguo), and Evan Liu built Silent Voice, a prototype that turns silently mouthed words into speech using a clone of the speaker's own voice.
The system combines three signals to recognize words as they form:
• Lips — a streaming adaptation of Auto-AVSR reads mouth movements through a webcam.
• Muscles — SVR-1, an EMG model developed by Frank Guo, reads facial muscle signals to distinguish visually similar words.
• Context — Claude follows both sides of the conversation to help resolve ambiguous words.
Every 100 ms, the system combines these signals and commits words when confident enough. Cartesia Sonic then speaks each word immediately, without waiting for the full sentence.
The team reports roughly 600 ms from muscle signal to generated voice.
Built for people who have lost their voice box, Silent Voice remains a hackathon prototype, demonstrated with a healthy speaker.
github.com/thaimtl/Silent-Vo…
#BackchannelSignals
We got tired of waiting for brain implants to give people their voice back.
So we built a 0.6-second latency voice decoder using four $20 muscle sensors, a webcam, and pure code in 36 hours at Cal Hacks. It placed 2nd overall out of 700+ projects.
It's built for throat cancer patients who have lost their voice box (laryngectomy). Because they still retain full movement of their lips, tongue, and jaw, the words are still there. Only the sound is gone.
How we hacked the latency wall:
1. Live Lip Reading: We took Auto-AVSR, a heavy, offline research model, and hot-wired it into a 100ms rolling loop using a fast greedy pass.
2. Muscle Fusion (EMG): We hooked up 4 surface muscle pads to an Arduino sampling at 1,000Hz. Our custom PyTorch model (SVR-1) reads the tongue and jaw movements that a camera can't see, immediately crushing lip-reading look-alike errors (like "bat" vs "mat").
3. LLM Forecasting: LLMs are usually too slow for live speech. While the caller is still speaking, Claude predicts what our speaker is likely to say next, tipping close calls toward contextually accurate words.
4. Expressive Voice: We streamed locked words into Cartesia Sonic to clone the speaker's voice in real-time. By mapping facial expressions straight into the voice API, a smile makes the output happier, and a clenched jaw makes it louder.
The data travels from raw muscle electricity → GPU inference → Claude pipeline → Cartesia voice generation → Twilio phone line in just 600 milliseconds.
Open-source, distributed on a ZeroMQ team bus, and orchestrated from a single laptop.
Repo: github.com/thaimtl/Silent-Vo…
Devpost: devpost.com/software/silent-…
Two fun bits of trivia about MOSS-Live's Shanghai-shot trailer:
1. Shanghai also provided much of the futuristic cityscape in Her (2013).
2. In The Wandering Earth, the sci-fi film based on Liu Cixin's work, the superintelligent AI is called MOSS. Its original designation is 550W. Turn that upside down and you get MOSS.
MOSS-Live is here. Our full-duplex, real-time voice AI can listen, speak, and get things done at the same time. 🎙️
While tasks run in the background, it can keep listening, seeing what you see, and talking with you.
• Natural conversation: It distinguishes real interruptions from “mm-hmm,” waits when you pause to think, stays out of side conversations, and gives brief acknowledgments while listening.
• Speech awareness & live translation: It tracks where it is in its spoken response, picks up after interruptions, and translates while you speak.
• Proactive help: Set rules for when it should correct mistakes, keep count, or remind you.
• Tools & background tasks: Check weather or stock prices, or let longer tasks run in the background without putting the conversation on hold.
• Multimodal perception: Use video calls or screen sharing to give it visual context. It can also recognize environmental sounds and describe voice qualities.
Try MOSS-Live ➡️ shorturl.at/96uVt
Silent speech recognition with less than $20 in hardware and just one hour of personal training data.
@_asashepard_ built a prototype that turns silently articulated words into open-vocabulary text, even with the mouth closed and lips still.
No audible speech, wake word, or fixed command list.
Asa reports 16% word error rate (WER), using about one hour of personal recordings plus synthetic data.
For context, earlier work by Gaddy & Klein used roughly 20 hours of single-speaker training data.
It's an early personal experiment, but suggests silent speech interfaces may need far less hardware and training data than previously expected.
asashepard.com/silent-speech
#BackchannelHardware
What if you could talk to your computer without making a sound?
The Subvocal Company (YC F26) is building glasses that turn nearly invisible mouth movements into text, using RF sensing rather than a microphone or camera.
Founded by @AarushYadav47 and Arnav Kumar, the startup has raised $1.3M from YC and Afore Capital.
The glasses send low-power radio waves through the face and measure how speech muscles change the signals. AI models then decode them into text.
The team reports open-vocabulary decoding at over 200 words per minute.
Four months ago, the project began with antennas taped to the founders' faces at a hackathon. Now they're working toward a wearable product, with plans to collect thousands of hours of RF data to improve generalization across users.
Reservations are open with a $199 deposit toward a $499 early-bird price.
ycombinator.com/launches/Wf9…
#BackchannelHardware
We’ve raised $1.3M from YC & Afore Capital so you never have to type or speak again.
Typing is slow. Talking into your computer is awkward. Subvocal fixes both
Our glasses give you the speed of speech without you making a sound or touching a keyboard.
It tracks your internal speech muscles with non-contact sensing and decodes silent, almost invisible mouthing into text at more than 200 words/min.
Pre-orders are now live at:
subvocal.co/preorder?utm_sou…
A voice AI avatar that listens, speaks, and lip-syncs in real time, right in your browser.
@victormustar built Ava, a demo that runs multimodal AI inference locally through WebGPU, bringing together:
• **@liquidai's LFM3-Omni** — speech understanding and generation.
• **@xenovacom's WebGPU kernels** — local inference in the browser.
• **@metacitizen's TalkingHead + HeadAudio** — facial animation and audio-driven lip-sync.
No cloud inference needed for the conversation model. Just a WebGPU-capable browser and a compatible device.
Try Ava: huggingface.co/spaces/townbo…
#BackchannelSignals
Meet Ava.
She listens, talks back and reacts in real time, right in your browser. All inference runs on your device, keeping your conversations private.
Powered by open source:
• LFM3-Omni: voice understanding & generation
• Hugging Face WebGPU kernels: fast, on-device inference
• TalkingHead + HeadAudio: avatar animation + lip-sync
Accuracy and cost at a fixed response deadline is the test that matters for voice agents too.
we think diffusion language models are a strong bet for the future of system 1 ai.
by system 1, we mean the fast response layer: answering a familiar question, completing an edit, or choosing the next action while the user is still in the flow.
the reason starts with inference. a dlm can refine many token positions in parallel. that creates an opportunity to get more useful work out of each model pass and fit capable behavior inside a smaller latency budget.
three results make this more than a nice idea:
1. llada 2.0 scaled diffusion to 100b total parameters. its optimized flash variant reported 535 tokens/s versus 256 and 237 for the two autoregressive baselines in its evaluation across four code and math benchmarks.
arxiv.org/abs/2512.15745
2. google reports over 1,000 tokens/s for diffusiongemma on a single h100. it is an experimental open model with 26b total parameters and 3.8b active, making this direction available for developers to investigate themselves.
deepmind.google/models/gemma…
3. inception reports 1,068 tokens/s for mercury 3, with quality comparable to speed focused models such as gpt 6 luna and claude haiku 5.5. this is a vendor claim, but it adds a commercial deployment signal alongside the open research.
inceptionlabs.ai/models
these numbers come from different evaluation setups. they establish promising examples, not a universal speed advantage. denoising takes repeated passes, and aggressive parallel decoding can trade away quality.
the test we care about is accuracy and cost at a fixed response deadline. include time to the first useful output and p95 latency under load. a huge tokens/s number means little if the user waits too long or the action is wrong.
our bet is a fast default path for familiar work, with deeper computation when the task demands it. system 1 is a product analogy here, and diffusion looks increasingly well suited to that role.
the opportunity is making useful intelligence fast enough to stay inside the interaction.
Can diffusion language models make real-time GenUI possible?
.@DiffuSpace_Labs has open-sourced Clover-35B-A3B-Preview, a 35B-total / ~3B-active MoE diffusion language model with block-parallel decoding.
Founded by @ikekong, @sansa19739319, and @JiachengYe15, the team reports around 680 tokens/s on a single NVIDIA GB300 (BF16, batch size 1) across internal agent workloads.
Speed matters when AI generates interactive interfaces and keeps adapting them as users explore.
One early example comes from @SigmaZ_official, Clover's exclusive GenUI ecosystem partner.
Together, they're exploring real-time interactive education:
• Inside a cell. Students can explore a 3D animal cell, click organelles, and switch between labels, cutaway views, and rotation.
• Across the solar system. An interactive Sun–Earth–Moon simulation lets students move through time, explore orbital alignment, and see why eclipses are rare.
The education demo is in closed beta at edu.tap8.ai/.
The teams are also working on joint post-training, targeting 2,000 tokens/s and broader real-time visual coding applications.
Clover's text-only weights are available under Apache-2.0.
Launch: diffuspace.ai/blog/clover-35…
Weights: huggingface.co/Diffuspace/Cl…
#BackchannelSignals
.@phonic_co raised $27.5M in Series A funding led by First Round and launched Phonic-1, as full-duplex conversation and real-time agent execution increasingly converge.
Its key feature is in-turn tool calling. Phonic-1 can start executing tools while users are still speaking, rather than waiting for the end of a turn.
• Demo — a 44-minute hotel booking conversation with 123 tool calls and mid-call language switches.
• Benchmark — Phonic-1 (High) ranked #1 on Artificial Analysis's Speech to Speech Index at launch, scoring 87.1.
• Developer access — API, Python/TypeScript SDKs, CLI, and no-code builder, starting at $0.07/min.
phonic.ai/blog/phonic-1
#BackchannelSignals
Introducing Phonic-1, the most advanced speech-to-speech model for agentic tasks — from @phonic_co, the speech-to-speech model company.
We’ve had a breakout year and raised a $27.5M Series A led by @firstround with participation from @Lux_Capital, @felicis, and Definition.
A Halloween voice AI game you can talk to.
@justsyun built a Rusty Lake-style escape room in a day. You're stuck in a room. Someone keeps knocking on the door every Halloween. And you can talk to the guy in the portrait.
Voice runs on @AgoraIO Conversational AI + @FishAudio TTS. Finding a voice like an old man telling a ghost story was harder than she expected.
#BackchannelSignals
Halloween’s coming so I made a rusty lake style escape room in a day.
You’re stuck in a room, someone keeps knocking on the door every halloween and won’t leave, and eventually you realize a kid is owed a candy and you can’t get out until he has it.
The big difference from actual rusty lake is you can talk to the guy in the portrait. Hopefully it pulls you into the story a lot more.
Voice runs on @AgoraIO conversational ai + @FishAudio TTS. Really love the voice I found on fish. Finding one that sounds like an old man telling you a ghost story was way harder than I expected.
To interrupt a voice agent on laptop speakers, the agent first has to stop hearing itself.
@johnrobinsn built MacVoice, a local voice agent for Apple Silicon Macs with an on-device voice pipeline. It turns on macOS's built-in echo canceller, the one FaceTime uses, so the agent's own playback is subtracted from the mic signal and you can talk over it without headphones.
On his M5 MacBook Pro speakers, the agent's voice at the mic drops from -21 dBFS to -78 dBFS.
github.com/johnrobinsn/macvo…
#BackchannelSignals
Voice agents that keep interrupting themselves usually have an echo problem, not a turn-taking problem... they hear their own voice. macOS ships a decent echo canceller. MacVoice uses it so you can talk right over it. storminthecastle.com/posts/m…
What if a voice AI could spend more time thinking without losing the rhythm of conversation?
NVIDIA researchers, including @CChen1436 and @ZhehuaiC, introduced Diffuplex, a full-duplex speech model that combines autoregressive (AR) interaction control with diffusion-based text generation.
The idea is to separate when to speak from what to say.
An AR policy continuously tracks incoming speech in 80ms steps, deciding when to listen, respond, or yield the floor. Meanwhile, diffusion generates and refines response tokens in parallel, without being tied to the audio frame rate.
The clever part is how it starts speaking: Diffuplex commits the first 8 tokens early and sends them to TTS while the rest of the response is still being refined.
The researchers report:
- 82.3 on VoiceBench in full-duplex mode without thinking, versus 58.1 for Nemotron-3 VoiceChat, the strongest full-duplex model in their VoiceBench table.
- 1,872 text tokens/s with thinking enabled, measured at batch size 1 on a single NVIDIA GB300 GPU.
- 239ms interruption latency and 396ms turn-taking latency, compared with PersonaPlex's 400ms and 70ms, respectively.
The thinking-mode throughput and full-duplex interaction results come from different configurations. Their combined performance remains to be demonstrated.
What interests us is the separation of conversational time from computational time: a voice agent could stay engaged in real-time interaction while allocating more compute to what it says next.
Paper: arxiv.org/abs/2610.17780
#BackchannelSignals
A character on your desk that looks like it's really there—without glasses.
@XYNbySony and @sony_jpn introduced ELF-MN1, a 6-inch glasses-free Spatial Reality Display for 3D characters, avatars, and objects. It connects to a compatible smartphone or Windows PC via USB-C, with an SDK for developers.
Launching in Japan on December 11, 2026, at an estimated ¥54,000 (~$340).
xyn.sony.net/en/news/2026-sr…
#BackchannelHardware
新製品リリース📢 6インチで小型の空間再現ディスプレイ『ELF-MN1』を発売
メガネやヘッドセットなしで、アバターやキャラクターが机の上にいるかのように裸眼で楽しむことが出来ます。
スマートフォンやPCと接続することで、まるでそこに実在するかのような新しい体験を提供します。
🔗sony.jp/CorporateCruise/Pres…
