Sonal Kumar retweeted
Our opening animation and blog didn't fully show the interesting dynamics for languages like CJK that have multi-byte characters.
In Chinese most characters take up 3 bytes, and when learning end-to-end from bytes it's not even trivial to get the model to chunk these into characters. Early in training this H-Net incorrectly sub-divides characters, but it eventually figures it out. Actually this was a prominent issue in the original H-Net - and even BPE doesn't get it right on rare characters. The blog has some examples!
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Introducing Cartesia Managed Agents: our end-to-end voice agent platform for enterprise.
#1 on @ArtificialAnlys Speech Agent Arena and @ServiceNow EVA-Bench, built by co-designing our voice models and harness as one system.
Check it out → cartesia.ai/blog/managed-age…
Really exciting work from @_albertgu and the Cartesia team!
One of the most interesting implications of H-Nets is that tokenization is not merely a preprocessing choice, but an architectural inductive bias that directly affects how models learn and generalize.
The results are particularly compelling: learned hierarchical chunking not only improves compute efficiency over fixed BPE tokenization, but also changes the relationship between pretraining loss and downstream generalization.
To me, the bigger research question is whether we can move toward architectures that learn their own representations and abstraction hierarchies directly from raw data, rather than relying on hand-designed discretization schemes.
This feels especially relevant for multimodal learning, where choosing the right representation granularity across audio, vision, and language remains a fundamental challenge.
A very interesting direction toward more general-purpose, end-to-end learning architectures. Worth reading both parts!
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
2
17
801
🎤🔥 Our third exciting keynote is LIVE!
We’re thrilled to have Gabriel Skantze take the stage for a fascinating talk on “Real-time Coordination Between Humans and Conversational Agents.”
🎤✨ Our second keynote is LIVE!
We’re excited to have Prof. Berrak Sisman from @jhuclsp take the stage! 🚀
If you haven’t joined us yet, now’s the perfect time to tune in. Don’t miss out on this exciting session! 🌟
63
🎤✨ Our second keynote is LIVE!
We’re excited to have Prof. Berrak Sisman from @jhuclsp take the stage! 🚀
If you haven’t joined us yet, now’s the perfect time to tune in. Don’t miss out on this exciting session! 🌟
We’re live! 🎙️ The SALMA Workshop (@SALMAworkshop26) is now underway at #EMNLP2026 (@emnlpmeeting) in Room F9–F10. Stop by for an incredible slate of keynote speakers and cutting-edge papers on speech and audio language models!
We’re live! 🎙️ The SALMA Workshop (@SALMAworkshop26) is now underway at #EMNLP2026 (@emnlpmeeting) in Room F9–F10. Stop by for an incredible slate of keynote speakers and cutting-edge papers on speech and audio language models!
Cartesia Managed Agents is #1 on the @ArtificialAnlys Speech Agent Arena, outperforming 20+ speech-to-speech systems on tasks like appointment booking and ordering.
We achieve frontier quality and naturalness by co-designing our speech models and harness in one vertically integrated stack.
Cartesia Managed Agents scores an Elo of 1115 on the Artificial Analysis Speech Agent Arena, 98 points above Cartesia's earlier voice agent product, Cartesia Line, with Task Success Rate up from 77.5% to 91.7%
Managed Agents is @CartesiaAI's fully managed platform for building and deploying voice agents. It is a cascaded system: Cartesia's own Ink Speech to Text and Sonic Text to Speech models run around an LLM of the developer's choice. We evaluated Cartesia Managed Agents with GPT-5.5 as the LLM and the Skylar and Daniel voices.
Key takeaways:
➤ Speech Agent Arena: Cartesia Managed Agents scores an Elo of 1115 (95% CI 1086-1144, 685 samples), 98 points above Cartesia Line at 1017 (95% CI 985-1049)
➤ Agentic tasks: On tasks that need tool calls, Cartesia Managed Agents scores an Elo of 1112
➤ Task Success Rate: 91.7% of eligible tool-calling conversations end with the correct final task-completing tool call
Congrats to the @CartesiaAI team!
See below for more details ⬇️
Sonal Kumar retweeted
Cartesia Managed Agents scores an Elo of 1115 on the Artificial Analysis Speech Agent Arena, 98 points above Cartesia's earlier voice agent product, Cartesia Line, with Task Success Rate up from 77.5% to 91.7%
Managed Agents is @CartesiaAI's fully managed platform for building and deploying voice agents. It is a cascaded system: Cartesia's own Ink Speech to Text and Sonic Text to Speech models run around an LLM of the developer's choice. We evaluated Cartesia Managed Agents with GPT-5.5 as the LLM and the Skylar and Daniel voices.
Key takeaways:
➤ Speech Agent Arena: Cartesia Managed Agents scores an Elo of 1115 (95% CI 1086-1144, 685 samples), 98 points above Cartesia Line at 1017 (95% CI 985-1049)
➤ Agentic tasks: On tasks that need tool calls, Cartesia Managed Agents scores an Elo of 1112
➤ Task Success Rate: 91.7% of eligible tool-calling conversations end with the correct final task-completing tool call
Congrats to the @CartesiaAI team!
See below for more details ⬇️
Sonal Kumar retweeted
🎙️ Meet the #SALMA2026 keynote speakers!
Join us at #EMNLP2026 in Budapest on Wed, Oct 28 for talks by Philipp Koehn, Berrak Sisman, Gabriel Skantze, Sebastian Möller, Piotr Żelasko & Julia Hirschberg.
📅 Program: salma-workshop.github.io/sal…
Sonal Kumar retweeted
Sonal Kumar, Sinan Hersek, Artem Dementyev, Mengzhen Pan, Ishan Chatterjee, Anurag Kumar, Ramani Duraiswami, Dinesh Manocha, Andrea Colaco, "SEA-LM: Egocentric Spatial Audio Understanding for Wearable Microphone Arrays" arxiv.org/abs/2610.05610
Audio LLMs are good at what they hear, but mostly not where it's coming from.
From my @Google internship: SEA-LM, egocentric spatial audio understanding for smart glasses.
One model → what, where, when, who (you vs. bystanders), and what was said.
arxiv.org/abs/2610.05610
How it works: any 4–9 mic glasses layout is beamformed into First-Order Ambisonics, encoded by FOACoder (pretrained on sound localization + wearer voice activity), and fed into Gemma-4-E4B alongside its native mono audio pathway.
Token dilution: long transcripts drown out the few coordinate and timestamp tokens in the loss. We upweight those tokens 5× with a spatio-temporal weighted cross-entropy.
On our held-out simulated eval set, vs. a SELDNet + Gemma-4 baseline (same LLM, data, curriculum):
• azimuth error 80.6° → 14.9°
• elevation error 27.4° → 12.8°
• fewer hallucinated sources
One checkpoint across 1,211 array configs (4–9 mics).
1
2
250
Scope: simulated, stationary scenes and two smart-glasses designs. Real recordings and moving sources are next.
Joint work between @gammaumd @umdcs @Google and @GoogleDeepMind with Sinan, @artem_dementyev @AcouIntel @dmanocha @ramani_d and others.
Paper: arxiv.org/abs/2610.05610
3
128
Sonal Kumar retweeted
🎉 Excited that “Not Every Rubric Teaches Equally” is accepted to NeurIPS 2026! 🎉
POW3R turns static rubric rewards into policy-aware training signals, emphasizing the criteria that actually separate good and bad rollouts for the current state of the model.
Link - arxiv.org/pdf/2605.20164
Our team at @ScaleAILabs has been cooking on some really exciting RL/post-training research lately. Check it out 👀
RGSD - arxiv.org/abs/2606.12507
Reward Hacking with Rubrics - arxiv.org/abs/2605.12474
Excited to share that TAC has been accepted at NeurIPS 2026.
TAC enabled timestamped audio captioning and speaker diarization in overlapping settings. TAC captions when combined with text-only reasoner achieves SOTA scores on most of the audio(-visual) benchmarks.
Work done at @Adobe with @pseetharaman, @urinieto, @justin_salamon, @ZeyuJin and others. Super exciting work.
More here: sonalkum.github.io/tacmodel/
Introducing SteerDuplex + SteerBench! We present a model whose speaking style can be steered mid-conversation, alongside the first benchmark designed to measure steerability in full-duplex dialogue.
Full-duplex voice models can listen and speak at the same time, but can they change how they speak when you ask them to?
Introducing SteerDuplex + SteerBench: the first benchmark that measures steerable full-duplex dialogue.
65.1% audio-steering APR vs 20.6% for the best open baseline. 🧵
6
253
NEW paper from Google
(bookmark it)
It's on advancing automated scientific review.
Just pay attention to the focus on agentic verification which is something I've been writing about recently.
AI is accelerating paper generation, but review is still mostly human throughput. The paper frames this as a systems problem and proposes four levels of AI-human collaboration for scientific evaluation.
It also discusses Google Paper Assistant as an early tool for scaling parts of the review process.
The important point is the direction of travel. AI-scientist workflows create verification debt. The review stack now needs its own agents, taxonomies, and human oversight patterns.
Useful read if you track AI research agents, automated peer review, or verification infrastructure for scientific discovery.
Paper: arxiv.org/abs/2606.28277
Learn to build effective AI agents in our academy: academy.dair.ai/
We researchers from US universities (UNC, UMD, Purdue, UW-Madison) are planning to host an audio-centric multimodal workshop at NeurIPS 2026, Atlanta. We would love to hear the community's opinion about this workshop through this form: forms.gle/on2wPomYhFJzk8X78. Thank you!
3
260
As speech and audio language models rapidly evolve, we need stronger conversations around architectures, data, evaluation, and safety for audio-first AI. SALMA brings together researchers working across audio LLMs.
Consider submitting or sharing with anyone in this space!
📢 Call for Papers is out - salma-workshop.github.io/sal…
We invite submissions to SALMA 2026: Speech and Audio Language Models Workshop, co-located with EMNLP 2026.🎙️
🗓️ Direct Submission: July 27, 2026
ARR Commitment: August 26, 2026
Please spread the word and consider submitting!
1
239




















