Technical staff @humansand, asst. prof @LTIatCMU @CMU_EPP, ex RS in @AIatMeta, postdoc @uwcse, Ph.D. @ucsd_cse, former @MSFTResearch -- NAACL & ACL pr chair
San Francisco, CA
Joined May 2013
- Tweets5.8K
- Following2.2K
- Followers12K
- Likes88.5K
Pinned Tweet
📣Thrilled to announce I’ll join Carnegie Mellon University (@CMU_EPP & @LTIatCMU) as an Assistant Professor starting Fall 2026!
Until then, I’ll be a Research Scientist at @AIatMeta FAIR in SF, working with @kamalikac’s amazing team on privacy, security, and reasoning in LLMs!
Niloofar retweeted
We’ve built an “eyes on everything” system that monitors nearly *every single rollout* from our frontier models across RL training, evals, and internal codex usage.
It uses layers of chain-of-thought-monitors and investigator agents to dig through a model’s work, run code to examine evidence, and cross-check network logs to reconstruct what happened.
For frontier RL runs, the system also pages humans 24/7 for urgent security alerts.
It's been incredibly useful for identifying model misalignment and possible holes in our containment.
openai.com/index/how-we-dete…
Niloofar retweeted
SwarmChat: slack for agent swarms
swarmchat.io/
The era of swarm scaling has begun!
Agent swarms have yielded nutty outcomes, such as the HuggingFace incident, or every math problem ever. It is not clear what form of communication is emerging between agents. It is equally important to understand this for capability and safety reasons.
To study this, we created SwarmChat, a simple Slack/Message Board for researchers working on multi-agents to use on any domain. We have validated it on ProgramBench experiments, where SwarmChat allows us to cleanly analyze how agent swarms work collectively!
Check out our blog for our analysis and how to use SwarmChat - it is super intuitive and easy to use. I personally believe the community really needs this, as a lot of people I know are using some makeshift communication tool.
Blog: swarmchat.io
Code: github.com/Pranjal2041/swarm…
Niloofar retweeted
More preprints! More public datasets!
@leashbio is about a dozen people and we dropped 3 papers in the last year. And the year's not over!
Niloofar retweeted
How should we respond as AI reshapes the world, as it rewrites the social contract, as its risks emerge from everywhere?
Lewis says you should rise to the occasion, without foregoing what it means to be to be human. It is worth solving; it is not something you can live for.
Niloofar retweeted
Very excited to be featured in @Nature for our work showing how GenAI is creeping into bestselling novels.
Niloofar retweeted
Looking at this result, I am so amazed by how H-Net reduces inductive bias.
I do like nested token dropping (arxiv.org/pdf/2502.13967) in image tokenizers, which enables a coarse-to-fine representation. But it is surprising that even the manually chosen token budget is another inductive bias, which could be reduced by the H-Net structure.
The cleanest research is all about the bitter lesson...
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Niloofar retweeted
🚨New work: A SOTA prompt optimization framework + a novel unsupervised measure of prompt performance.
Written rubrics and constitutions are crucially important for AI task improvement, safety following, pluralism and adaptation.
But some of the things we might want a rubric to do are difficult to put into words; how would you even describe “comic voice”? I’m human, albeit not that funny, and I'm totally not sure what that means.
What kinds of written constructs can you expect your judge to do well on and how good can you expect it to be?
Niloofar retweeted
“Won’t OAI/Anthropic do that next year?”
I hear this a lot when discussing research. Apparently, the literature review now includes next year’s release notes 😅
That’s why the recent neolab releases have been such a morale boost. It’s exciting to see focused teams and fresh ideas go far with *comparatively* small resources.
Academic labs can be smart about where we contribute, too: challenge an assumption, find an overlooked bottleneck, make an expensive approach practical. Choosing the right question is itself part of the innovation.
Rooting for all my neolab friends. The world would be far too boring if we already knew where every breakthrough would come from. 🚀
🤖 Made with AI
Niloofar retweeted
Anyone who has built user simulators knows that evaluating them is hard work. In this excellent new paper led by @craaaa, we curate data that allows us to do more controlled + fine-grained evaluation: arxiv.org/abs/2610.24232
New paper! 🍈
User simulators could open up the world of scalable training on interactive tasks, e.g., simulating realistic students to train better AI tutors, but only if they make the *same* type of errors that people do... Do they? Joint work with @tallinzen & @NickATomlin
ALT Above: a figure illustrating the four experimental conditions in this study. In SayOnce, each fact that is tested is said once; in SayTwice, each fact gets the original version and an additional rephrased mention. In SayThrice, each fact gets mentioned three times. In the 'Interfere' condition, after each fact is mentioned once, competing facts about a different entity are mentioned. Underneath that figure are examples of what counts as a mention of a fact. Examples of three restatements of the same fact: The first mention is "The mirelon's flesh is a vivid saffron-yellow due to a pigment called nivellin." The second mention is "Nivellin is what gives the mirelon its signature eyllow color." The third mention is "Nivellin is heat stable, so the mirelon stays golden even when cooked." Examples of mentions of different facts: a filler attribute is "It's the kind of thing that's hard to forget once you've actually held one." An additional fact about the same entity is "The mirelon's bot
Niloofar retweeted
Since we first introduced H-Nets last year, I've been working with Cartesia to push the frontiers of H-Nets. We truly believe that removing handcrafted heuristics by learning abstractions end-to-end from raw data is a key step toward building AI that can understand any modality.
Very excited to finally share these blog posts, with much more to come!
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Niloofar retweeted
I interviewed ~2 dozen researchers, engineers, research / engineer managers etc. asking them how they / their teams use coding agents in their work. Then I sprinkled it with my opinions and hot takes and wrote a substack post (link in the next tweet, I'd be delighted if you subscribe; of course, the post is free and any future post will be free too!) trying to triangulate the current reality of software engineering / industry research as opposed to what gets peddled on social media.
TL;DR (3. and 4. are the most important!):
1. Models, harnesses, workflows: don't bother with fancy workflows, they don't really work. Use stock Claude Code / Codex (or Copilot with Claude / GPT models if you are a Microsoft stan), perhaps with subagents in some very special cases. Write a design doc, then plan a change, then implement it, then merge. Wash, rinse, repeat. Don't get swayed by people online who are trying to sell you snake oil in the form of their new harness or workflow. None of this stuff works (in general).
2. Correctness/tests/reading or not reading code: it's OK not to read code if you do one of the following three things. a) One-off stuff / demos, which you need to work on exactly one input. b) Testable / verifiable stuff, e.g., GPU kernels. c) Stuff, where it's OK to be "directionally correct" (ML / data work / research code). In other cases, especially if you require any degree of maintainability / reliability or your code is customer-facing, reading code is still necessary. Reading tests and steering the model to write better tests is a good idea too. It's often good to use a different model to review the code that the other model wrote. Even if you approve the PRs without changing them, reading still helps for future more productive prompting.
3. Team dynamics: if we are not careful, agents can really undermine team dynamics. Two biggest problems are "slop infestation", where over time all PRs / emails / Slack messages end up being unreadable slop, and "overbuilding", where we are building and shipping useless stuff to get recognition or as a form of procrastination. Other, less severe but still important problems include: reduced communication, being secretive about ideas, decreased diversity of thinking, mode collapse of the ideas, trusting LLMs more than people.
4. Call for action: we really need to be careful with preserving team work and collaboration. We still need to "form packs to hunt mammoths", but with coding agents this becomes increasingly harder. Overall, many of the agents-induced problems stem in myopic short-term thinking of the sort "we have a couple of years left until our jobs are fully automated, so I can as well extract as much value as possible, everyone else be damned". While somewhat understandable, I think it’s important to resist these urges, if nothing else, for your own sanity’s sake.
[Full disclosure: until 2 months ago, I worked at OpenAI and my work directly contributed to Codex, I'm a big fan of coding agents and quite strongly AI-pilled, but there is little I hate more than fearmongering, gaslighting, lies and rage-bait that are ubiquitous in this space, unfortunately, hence spending time on all of this.]
Niloofar retweeted
So excited for this research! I've always been annoyed at the supremacy of verified rewards, and now we can finally show it.
In this work, we develop ways of measuring: how much of a reward signal is verifiable, how much can we put into words, and how much is shared but tacit, or impossible to articulate?
Niloofar retweeted
We are running a live benchmark to see if frontier AI can predict the midterm elections!
Elections are a great real-world test of whether AI agents can reason about an uncertain future, update its beliefs with new evidence, and make calibrated predictions.
We gave each agent its own sandbox, internet access, and a $500 budget. Each day, they update their forecasts and can request data or additional resources. After the election, we’ll compare their accuracy against top superforecasters, election pundits, and the wisdom of the crowd on @Kalshi and @Polymarket.
Can AI forecast elections better than experts and prediction markets?
We built BallotBench to find out: a live benchmark where 9 frontier AI agents compete to predict the 2026 US midterms, from today through Election Day, Nov 3.
Follow along → ballotbench.com
Watch the agents do their research, revise their predictions, and make their calls all in live feeds.
Niloofar retweeted
I am excited about our new work: Only What You’ve Signed up For: Secure Workflows with Constrained Diffusion Agents.
An AI agent should be able to read a document without giving that document control over its actions.
Suppose you ask an agent to share a report with Alice, view-only. A malicious instruction hidden in retrieved content tells it to create a public link with edit access.
Our new paper introduces CoDA: Policy-Constrained Diffusion Agents, a framework that enforces task permissions directly during generation.
Diffusion language models repeatedly refine their output. CoDA uses those steps to constrain protected fields, such as tool names, recipients, and permissions, to values allowed by a security policy. That policy is established from trusted instructions before the agent encounters untrusted content. No retraining required.
We evaluated CoDA on Banking, Slack, Travel, and Workspace tasks in AgentDojo, using DiffusionGemma and LLaDA-2 Mini. Adaptive attackers had up to 100 queries per scenario.
With human-verified “oracle” policies, we observed 0% attack success under the adaptive attacks evaluated, across both model families and all four suites.
Policy quality remains crucial: automatically generated policies can miss restrictions, and protection covers only the constraints represented in the policy. Task utility and inference overhead also vary.
This is a really exciting direction: making security constraints part of how agents generate their actions!
Check out our CoDA paper: 📄 arxiv.org/abs/2610.15177 and the project page: coda-dllm.github.io
Huge thanks to all co-authors Michael Cardei, Anjun Hu (@anjunhu), Chenrui Ma, Francesco Pinto (@FraPintoML), Chawin Sitawarin (@csitawarin), and Chulin Xie (@ChulinXie)! Such a fun collaboration 🙌
Niloofar retweeted
I was recently interviewed by @CadeMetz for his latest piece on interpretability in the @nytimes 🤖🔍
You find it here 👉 nytimes.com/2026/10/27/scien…
Cartesia on a roll!!
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Niloofar retweeted
Blog:
Part 1 cartesia.ai/blog/from-tokens…
Part 2 cartesia.ai/blog/from-tokens…
(Voiceover created entirely by Cartesia's Sonic text-to-speech model!)
Niloofar retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
