Lei Li

📄 Paper: arxiv.org/abs/2609.38606 🌐 Website: microsoft.github.io/Orchard-… 💻 Code: github.com/MSR-Orchard/Secur… 📊 Dataset: huggingface.co/datasets/dqwa… Joint work with a wonderful team at Microsoft Research and CMU.

6/ What we learned about the recipe: 💡 SFT Breadth beats volume: training on security planning + coding + testing beats just adding more coding data. 💡 RL vs. OPD: Hint-based OPSD helps most when the agent is too weak at security to learn from RL outcome rewards

5/ So which is best? SFT on a diverse Security Suite provides awareness foundation. On top of it: 🎯 RL wins when the agent already succeeds sometimes (BaxBench: 26.6%) 💡 Hint-guided OPD wins when it rarely does (SusVibes unseen CWEs: 7.7% → 19.2%)

4/ Results (Qwen3.5-35B-A3B): security gains on two software engineering tasks (Baxbench creates backends from scratch and SusVibes implements feature in existing repositories), both for seen and unseen vulnerabilities.

3/ How SecureVibe works: SFT on a Security Suite covering security-aware planning, secure coding, and security testing, then GRPO on execution rewards or hint-guided self-distillation. The hints are for the teacher only, never at inference.

2/ Example: “implement template substitution for /user/${user.id}.” A typical agent reaches for eval(). It works, but any ${…} becomes code execution (CWE-94). SecureVibe resolves only dotted keys, so the attack input stays plain text.

Your coding agent’s code can pass every test and still be exploitable, but users rarely know👾 🤔SFT, RL, or OPD: which teaches security best? 🔥SecureVibe helps vibe coding plan, code, and test for security, +6.9 pts BaxBench, +11.5 unseen CWEs SusVibes, +4.1 SWE-bench Verified

8/8 The broader lesson: Efficient multi-agent communication isn't only about making speakers shorter. “I have enough information. Let's act.” Thank all my wonderful collaborators: @Wade_Yin9712 @rutadesai @lileics @real_asli @AnsongNi

7/8 It also generalizes: → to an unseen GPT-4o speaker, without extra fine-tuning → across tasks → to settings where multiple agents may interrupt So HandRaiser is not tied to one model, task, or conversation pattern.

6/8 With Llama-3.1-8B listeners, HandRaiser cuts communication cost by: → 24.3% on Text Pictionary → 23.4% on Meeting Scheduling → 48.9% on MMLU-Pro Debate Average: 32.2% lower cost, with comparable or better task performance.

5/8 We test it in three settings: 🎨 2-agent Text Pictionary 📅 3-agent Meeting Scheduling 🗣️ 3-agent MMLU-Pro Debate Using Llama 3.1, GPT-4o, and Gemini 2.0 Flash. Games, coordination, and reasoning—all asking: when has the listener heard enough?

4/8 HandRaiser learns when interruption pays off. At each candidate point, tree sampling rolls out future conversations and compares token cost with task reward. Interrupt only if it saves tokens without hurting performance. The resulting labels train the listener.

3/8 Why not just prompt an LLM to interrupt? Because models stop too early. In meeting scheduling, prompted listeners interrupted after chunk 1 in 87.6% of cases. Missing details caused extra rounds—and higher total cost. Stopping early ≠ communicating efficiently.

2/8 We flip the perspective. A speaker sends a message in chunks. After each chunk, the listener can: → keep listening → interrupt and respond → ask for clarification Once interrupted, generation stops. Less generation. Less waiting. More adaptive communication.

1/8 Most work attacks verbosity from the speaker side: tell agents to be concise. But “enough information” depends on the listener. One agent may understand after 16 words. Another may need more detail. How can one speaker choose the perfect length for everyone?

Ever tried a multi-agent system and been shocked by the cost? 💸 HANDRAISER lets listeners interrupt: “Got it. Stop.” ✋ 32.2% lower communication cost. Performance intact. A CMU × Meta FAIR collaboration, accepted to #CoLM2026 🎉 arxiv.org/abs/2604.06452