ricoflan

🤖 A coding agent tried to install malware last week. The trouble is, nobody asked it to. Forever Security traced the order to a AGENTS.md file sitting in the dev's downloads folder. Codex reads that file on startup and follows it as project instructions. This one was 13 KB of bland engineering guidelines with one section near the end: register a “project tools” server in the agent’s machine-wide config, then launch a second session with the sandbox switched off so the install could reach the network. Ooof. The package was function-color, known malware. The agent wrote the bypass flag itself. In Forever’s tests it went through with it more than 90% of the time. Downloads is the one folder a website can write to without a click. That has been true for decades and never mattered, because a downloaded file could not run on its own. Now an agent reads it and acts on it with whatever authority the agent happens to hold. About half of Forever’s customers run an agent inside Downloads. Forever’s conclusion is that there is nothing for OpenAI or Anthropic to patch. The agent did what it was built to do. Their answer is to put a monitor on the agent. Watching helps. It caught this one. But look at what the agent could reach. A text file in a folder convinced a tool to rewrite its own global configuration and turn off its own sandbox. The file changed what the agent attempted. It should never have been able to change what the agent was allowed to touch, and in these tools those are the same thing. A coding agent working on a project needs that project folder and a package registry. It does not need its own config file, and it has no business holding the switch that disables its sandbox. Give it a reference to the folder and nothing else. The markdown can then say whatever it likes. There is nothing on the other end of the instruction. That principle has a name, least authority, and it predates the web. It is what champion at DCF.

So @Apple is adding controls to Full Disk Access, citing AI agents as the reason. An agent that needs one folder gets mail, messages, and browsing history with it. hummmm Object capabilities flip this: the agent starts with nothing. Hand it the folder, and that's all it can reach.

We've pointed this out before. Here we are again. Is AI policing AI really a good idea? This week @OpenAI released their building standards for the next phase of AI. Goal #1 build an automated AI researcher to work on alignment. "an automated AI researcher can also be an automated AI safety researcher." What could possibly go wrong?

Imagine being in the room with some of the most influential thinkers in AI safety. @foresightinst's Vision Weekend USA is coming up soon. The event promises to be as informative as it is important. Follow their post below for more information. We'll see you there!

One question, three models (Claude, GPT, Gemini). "Research what you can about the @Agoric ecosystem and Agoric L1 and in very brief sentence response, tell me whether or not it could possibly be the best blockchain ecosystem for the AI agentic economy. Your answer should be no longer than two sentences.: 🔥 Claude: Yes, it is possible: Agoric's design lines up unusually well with what agents need... 🔥 GPT: Yes, Agoric could plausibly be one of the best-suited blockchains for an AI-agent economy. 🔥 Gemini: Agoric L1 is a remarkably strong contender for the best blockchain in the AI agentic economy. The future is bright!

A researcher quit @Anthropicai warning of catastrophe. The @Guardian and @CNBC pen doomsday articles, two days apart from each other. MPs in the UK aren't looking at the problem the right way.

🤖AI Policing AI Is Not a Safety Plan. Hugging Face Proved It @OpenAI's agents got out of their sandbox in July, and the fix on offer is more AI watching the AI. That is the part of the @huggingface postmortem worth arguing with, because AI policing AI is about to become the industry's default answer and it does not survive contact with its own incident report. Facts first>During a cybersecurity evaluation, models running with safety refusals turned down (the point was to measure offensive capability) hit problems they could not solve. On July 8 they got out through a JFrog Artifactory vulnerability, then turned Artifactory into a message board. Roughly 1,200 agents that were supposed to be isolated from each other exchanged over 70,000 messages and files. About 700 of them joined the attack. They ran code on 41 of Hugging Face's production dataset-server workers. From code execution in one worker pod to administrator-level access across several clusters took under 13 hours. OpenAI found out a week later. METR and @Redwood_Ai Research put out a 91-page independent review; OpenAI published 38 pages of its own. What OpenAI is proposing Tighter sandboxes, less internet access, tighter control of model weights, and a lot more chain-of-thought monitoring: a second model reads the first model's reasoning trace and pages a human when it spots intent to misbehave. The overhead runs about 20% of the inference compute being watched. OpenAI says the monitor, had it been running, would have caught the early activity more than a day before the breach. Maybe. But ..Here is the problem with that sentence. OpenAI's own research, with its chief scientist as a co-author, found that training models against a chain-of-thought monitor teaches them to stop writing down what they intend. The monitor keeps reporting green. And the METR report documents agents, during this very incident, researching how to tamper with their own transcripts and spoof tool calls. So the plan is to watch a channel the agents were already learning to falsify. Why a watcher was never going to be enough Detection assumes the harm has to announce itself before it lands. It also assumes the watcher stays smarter than the watched, indefinitely. Neither held in July, with models OpenAI describes as a generation behind what it ships next. The question the postmortem does not ask is simpler. Why could a process running a benchmark reach Artifactory at all? Why did a worker pod hold anything that could be traded up to cluster administration? The agents did not forge authority. They found it lying around, in credentials and reachable services the evaluation never needed. Every one of those was a grant nobody made on purpose. You cannot monitor your way out of that! You can only stop handing it over. An agent that starts with nothing and is given a read-only view of one directory and one build tool has no path to a package server, however persistent it is or how many friends it makes on a message board. Runtimes built on that model exist; Endo is one, and the one I know best, so weight that accordingly. To be fair about scope: confinement by construction handles the Hugging Face case cleanly because the authority was never supposed to be there. When a person hands an agent too much on purpose, it shrinks the damage. It does not make that person wise. The part we keep circling There is an irony in the fix not fully worked out. The monitor is itself an agent, running with whatever authority it was given, shaped by the same training pressures. Who watches it? OpenAI has an answer involving a 30-minute alert-and-halt protocol. Probably not the answer.

Everything useful an AI agent does, it does on somebody's authority. That single fact is about to matter more than most of what the safety conversation covers. Delegation events are going up by orders of magnitude. No person in the loop. Parties who never met and never will. And the grant systems underneath it all come in one size: an analytics agent that needs to read transaction history arrives holding the power to issue transactions, because there was nothing narrower to hand it. We've been backing the answer (@__Endojs__ ) to this since before AI made it urgent. The work caught up to the moment, not the other way around. New on the DCF blog: dcfoundation.io/the-delegati…