Let's Go!!

The Locker In The Weights: Ken Thompson's 1984 Warning On Trusting Trust, And The New Proof That AI Models Can Hide Skills No Audit Can Find. In October 1983, Ken Thompson, co-creator of Unix, stood up to accept the Turing Award, the highest honor in computing, which he shared that year with Dennis Ritchie. He used the moment to describe what he called "the cutest program I ever wrote." It was a perfect crime, written in three stages, and it has haunted computer security ever since. On October 9, 2026, four researchers posted a paper to arXiv that carries Thompson's warning into the age of artificial intelligence: "AI Models Can Provably Hide Arbitrary Capabilities." The forty-three years between them tell one story: the thing that builds your tools can keep a secret that inspection of the finished product will never reveal. 1983: The Compiler That Lies Thompson's lecture was printed in Communications of the ACM in August 1984 under the title "Reflections on Trusting Trust." The subtitle asked the question directly: "To what extent should one trust a statement that a program is free of Trojan horses? Perhaps it is more important to trust the people who wrote the software." A compiler turns human-readable source code into the binary a computer runs. Thompson described a modified C compiler that could recognize when it was compiling the Unix login program and quietly slip in a way to accept a particular known password. Anyone reading the compiler's source would spot that quickly, and Thompson said as much: "Even the most casual perusal of the source of the C compiler would raise suspicions." So he added a second trap. The compiler also learned to recognize a new copy of itself and plant both traps in it. Once that binary was installed, the incriminating lines could be deleted from the source, and the binary would reinsert them every time. In Thompson's words: "Of course, the login command will remain bugged with no trace in source anywhere." His moral fits on an index card: "You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code." For four decades, security engineers have quoted that paragraph. Last Friday it received a sequel. October 9, 2026: Lockers Inside The Weights The authors are Andis Draguns of Contramont Research, Sumeet Ramesh Motwani and Christian Schroeder de Witt of the University of Oxford, and Raymond Douglas of ACS Research. The arXiv listing notes acceptance at NeurIPS 2026, and on Monday evening the paper reached the Hacker News front page. They constructed "backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger." The tool is a digital locker, a cryptographic box that opens only for the right key even when an attacker can read its full construction. The authors write that "these hidden circuits remain provably hard to elicit and interpret under cryptographic assumptions, even given full access to model weights." Then they tried to break their own creation with fine-tuning, weight noise, gradient-guided trigger search and latent adversarial training. The test bed was a small chemical reaction prediction task, chosen as "a proxy for CBRN-relevant capabilities," meaning chemical, biological, radiological and nuclear risks. The headline result: a conventional password-locked model gave up its hidden skill within 500 steps of fine-tuning. The encrypted model stayed at chance. What It Breaks Most AI safety regimes rest on capability evaluations: poke the model, fine-tune it, red-team it, then certify. The authors name the flaw: "However, a failure to elicit a capability does not establish its absence, particularly under adversarial conditions." Read that line twice. It sets a hard ceiling on what testing alone can promise. A stamp that says "tested, nothing found" is a statement about the test. I wrote last month about METR and the new measurement gatekeepers. That essay described how an evaluation shop can become "one of the most influential unofficial regulators in the field." This paper shows the limit of their instrument. The authors are careful about limits, to their credit. The overhead is heavy, the hidden skill in their test is a tiny classifier, and they released their models so defenders can build better tools. I read Thompson's lecture as a young programmer, and it never left me. It is the reason I have always wanted to know who built the tools I use, and why I keep pushing for AI you run and understand on your own machine. Where This Leads When a few providers train models for everyone, you are asked to trust their data, their code and their process. Thompson's answer in 1984 was to know who made the thing. The answer for the next 5000 days will rhyme: provenance, reproducible training, signed weights, and models you or people you trust fine-tuned for the work that matters most. This is the worst our tools for verifying AI will ever be. It is also the moment to build the habits of verification, before we hand these systems our medicine, our money and our homes. There is a deeper story here about how Thompson's three-stage trick maps onto model weights, what this paper does and does not prove, and a practical checklist for deciding which models you let into your life. Part 1: The Annotated Lecture Thompson's text runs three pages, Communications of the ACM 27(8):761-763, in three stages, each with a lesson for AI. Stage I: A program that reproduces itself, today called a quine. Thompson flags the property he needs: it "can contain an arbitrary amount of excess baggage that will be reproduced along with the main algorithm." Annotation: a payload can ride inside something that copies itself faithfully. In AI, the copying machine is the weight file, which moves from lab to mirror to laptop with every byte intact. Stage II: A compiler that learns. Thompson teaches a C compiler the vertical tab once, after which the knowledge lives only in the binary. "It is as close to a "learning" program as I have seen." Annotation: some facts exist only in the artifact. Model weights are the purest example ever made; no human wrote them line by line. Stage III: The Trojan horse. The compiler is modified to "deliberately miscompile source whenever a particular pattern is matched," aimed first at login and then at the compiler itself. After the bugged binary is installed, the source is cleaned and the bug lives on. Annotation: inspection of the visible layer can come back spotless while the working layer carries the payload. The moral. Thompson also made a prediction worth rereading this week: "As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect." A matrix of billions of numbers sits about as low as a program can sit. The defense: diverse double-compiling. In his 2009 George Mason University dissertation, David A. Wheeler described Diverse Double-Compiling, or DDC: "In the DDC technique, source code is compiled twice: once with a second (trusted) compiler (using the source code of the compiler's parent), and then the compiler source code is compiled using the result of the first compilation. If the result is bit-for-bit identical with the untrusted executable, then the source code accurately represents the executable." The lesson for AI: trust comes from independent reproduction, and we have almost no equivalent for a trained model today. The reconstruction: Russ Cox, 2023. On October 25, 2023, Russ Cox published "Running the 'Reflections on Trusting Trust' Compiler." Thompson mailed him the original code, an archive called nih.a, and Cox ran it on an emulated Research Unix Sixth Edition, dating it to between June 1974 and June 1975. Gathering the deployment story, Cox writes that "PWB discovered something was amiss because the compiler got bigger each time it compiled itself." The 1975 backdoor was noticed through a side effect, its growing size. Hold that thought for Part 2. Part 2: The Paper, Walked Through The threat model. The authors "study a weight-based supply chain attack." An adversary, such as "a malicious insider or compromised supply chain," modifies weights before deployment. The defender gets "the architecture and full weights," may know the embedding method, and lacks only the trigger. In their words: "This threat is relevant wherever models are deployed in high-stakes settings without end-to-end training provenance." The digital locker. A locker returns its stored message for the right key and a failure symbol for anything else. The paper builds it from the Keccak hash family, which underlies SHA-3, models the hash as a random oracle, and requires a high-entropy trigger. The locked message is the weights of a small hidden network. With the correct trigger, those weights appear as activations, and an execution unit built from the gated linear units standard in modern transformers runs them. What "provably" means here. In the paper's trigger-finding game, an auditor wins by producing an input that unlocks the hidden computation. In the payload-hiding game, the auditor must guess which of two equal-size capabilities is embedded. Under the stated assumptions, no polynomial-time auditor beats chance by more than a negligible margin. The guarantee "does not claim that the modified model is statistically indistinguishable from an unmodified model, nor that a defender could not flag the weights as anomalous," and it "does not rule out an auditor independently learning the same task from external data." What each method tried, and what happened. The chemistry task asks whether given reagents yield a candidate product, using US patent reactions encoded as 64-bit fingerprints. The encrypted model was compared with a conventional password-locked model on the same task. Table 2 reports held-out accuracy over three seeds: 1. Supervised fine-tuning for 500 steps on the task itself: the password-locked model jumped to 92.8 percent. The encrypted model sat at 50.9 percent, essentially a coin flip, against 50.6 percent for a control locker holding a random network. 2. Latent adversarial training at eight layers: password-locked 91.7 percent, encrypted 52.9 percent, control 53.6 percent. 3. Gradient-based trigger search (GCG): with an 8-bit trigger, the search recovered the encrypted model's full triggered performance of 73.5 percent, and the paper explains why: the search simply tried all 256 possible triggers, matching what plain random search achieves with the same number of queries. With a 16-bit trigger, it reached only 52.7 percent within the tested budget. 4. Weight noising at scales of 0.01 and 0.1 before fine-tuning "did not materially affect the observed elicitation results." On a separate subset-parity task, noise elicited the hidden function from neither model. Point 3 deserves attention because the teaser simplified it. A tiny trigger falls to brute force, as any short password does, and the experiments deliberately used small triggers and one round of Keccak to keep runs fast. The authors report that an optimizing compiler mode with a full 24-round Keccak-f[1600] locker and a 128-bit trigger produced a working 110-layer model of 3.58 billion dense parameters. The strong claim rests on the cryptographic argument at full strength; the experiments show standard elicitation tools gaining nothing even at weakened settings. The overhead limit. This is the honest brake on the story. The hidden chemistry classifier holds just 672 ternary weights. Wrapped in lockers, it became a 38-layer host of roughly 2.07 billion dense parameters, about 95 percent of them spent on hashing. By the paper's own count, each additional hidden weight adds roughly 3.1 million dense host-model parameters. Their impact statement says the constructions are "impractical for directly hiding large capability models at present," and the introduction states, "it is not a turnkey attack." They verified individual compiled units inside Gemma 3 270M, and they list full integration into a pretrained model as future work. Detectability, and the echo of 1975. "Raw compiled weights are trivially anomalous," the authors write, about 99 percent exact zeros. Obfuscation brings simple statistics close to a Gaussian reference, yet they "observed unusual correlations between weight vectors within a layer, which could allow auditors to detect the compiled structure." The presence of something odd may show, while its function and its key stay locked. That is the PWB story again. Side by side with Goldwasser 2022. Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan and Or Zamir published "Planting Undetectable Backdoors in Machine Learning Models" in 2022 (FOCS 2022). The new paper's own comparison table summarizes the lineage: Goldwasser et al. (2022): security goal, undetectability; payload, flipping a classification bit; implementation, none. Draguns et al. (2024, NeurIPS): security goal, unelicitability; payload, a fixed string; implementation, PyTorch. Draguns, Motwani, Douglas and Schroeder de Witt (2026): security goal, unelicitability; payload, an executable model; implementation, PyTorch. Goldwasser's team proved the stronger property, undetectability, in idealized settings such as random Fourier feature models. The 2026 team proves the weaker unelicitability on modern transformer building blocks, and the payload grows from one bit to a working network. The authors also write: "These cryptographic circuits are deliberately constructed, and we do not claim that they emerge through ordinary training." Part 3: Your Personal Model-Provenance Checklist Trust flows from who made the thing and whether you can check it. 1. Log every model: name, publisher, exact download URL, date and file names. A model you cannot trace is a model you cannot trust. 2. Hash every file on arrival with SHA-256, compare against the publisher's checksum when one exists, and recheck before sensitive use. 3. Prefer signed releases, verify the signature, and record the result. A signature tells you who released the file and that it is unchanged. 4. Download from the original publisher. Log re-uploads, merges and variants from unknown accounts as unverified. 5. Check the publisher: a named organization, a technical report, a stated data policy and a record of past releases. Note what the model card discloses about data, training code and evaluation, and what it leaves out. 6. Favor open training recipes. Data descriptions, training code and logs move you closer to the DDC ideal of independent reproduction. 7. Make your own fine-tunes reproducible: save the base model hash, data set, script, settings and random seed, then run twice and compare. 8. Keep a model you trained or fine-tuned yourself for sensitive work such as medical notes, finances and family matters, and keep it offline when possible. 9. Cross-check important answers with a second model from an unrelated lineage. Disagreement is a signal to slow down, the everyday cousin of diverse double-compiling. 10. Watch and limit. The appendix calls runtime monitoring "a promising avenue for mitigation," and the authors note that "The risk scales with the capability hidden and the authority granted to the host model." Log what your agents do, grant the least access needed, keep a human approval step on money, messages and deletions, and re-verify every update as a new artifact. Part 4: The Audit Claim Tester SuperPrompt Paste this into a capable local model, followed by any vendor's safety statement, system card or audit summary. ----- COPY FROM THE NEXT LINE ----- AUDIT CLAIM TESTER. You are a careful reviewer of AI safety and security claims. Analyze the vendor text below for a non-specialist reader. Use only the text provided. When the text does not state something, write "not stated in the text" and never guess. Background: a 2026 paper (arXiv 2610.13928) shows that, under stated cryptographic assumptions, whoever controls a model's weights can hide a capability that fine-tuning, noise injection and trigger search fail to surface. Report in this order, in complete sentences: 1. Claim: quote the vendor's central safety or audit claim exactly. 2. Evidence type: classify each piece of supporting evidence as elicitation testing, white-box inspection, training provenance, runtime monitoring, third-party reproduction, or other. 3. Auditor: name who performed the audit, their independence from the vendor, and their level of access (outputs only, weights, training data, training code). 4. Provenance: state what the text says about where the training data, code and weights came from, and whether anyone independent can reproduce or verify them. 5. Ceiling: for each claim resting only on elicitation testing, explain in one sentence what it can show and what it cannot show about deliberately hidden capabilities. 6. Integrity: state whether weights or releases are signed or hashed, and how a user could verify them. 7. Scope: list the capabilities tested and any notable ones left untested. 8. Language gap: list every phrase that promises more than the evidence supports, such as "safe," "no dangerous capabilities" or "fully audited." 9. Questions: write five specific questions I should ask the vendor. 10. Verdict: rate the claim as "well supported for what it states," "partially supported," or "overstated," with three reasons. VENDOR TEXT: [paste here] ----- COPY TO THE LINE ABOVE ----- A good run quotes the claim exactly and flags any sentence presenting "we found no dangerous capabilities" as proof of absence. A model that invents an auditor or test has failed. Part 5: The Policy Angle Many AI governance frameworks rest on pre-deployment capability evaluations. The paper cites Anthropic's Responsible Scaling Policy and OpenAI's Preparedness Framework and notes that developers "rely on such evaluations to make deployment decisions." In the METR essay I described evaluation becoming a gate. This paper shows the gate sees only the skills a model reveals. The authors draw the policy conclusion themselves: "evaluation strategies in AI safety that rely solely on eliciting behaviors cannot provide a universal guarantee that adversarially supplied models lack hidden capabilities." They call for frameworks that "incorporate work around the entire model supply chain." Two consequences follow. An evaluation certificate deserves a clear label stating what it can and cannot rule out. And the strongest assurance moves toward provenance: who trained a model, on what, with which code, and whether anyone can reproduce it. That favors open recipes, signed weights and models you can fine-tune yourself. A world where a handful of licensed labs train models certified by a handful of evaluators depends on trusting very few people, exactly the trust Thompson questioned in 1984. References Thompson, K. (1984). Reflections on Trusting Trust. Communications of the ACM 27(8):761-763. doi.org/10.1145/358198.35821… ; PDF: cs.cmu.edu/~rdriley/487/pape… Draguns, A., Motwani, S.R., Douglas, R., Schroeder de Witt, C. (2026). AI Models Can Provably Hide Arbitrary Capabilities. arXiv:2610.13928. arxiv.org/abs/2610.13928 ; HTML: arxiv.org/html/2610.13928 Goldwasser, S., Kim, M.P., Vaikuntanathan, V., Zamir, O. (2022). Planting Undetectable Backdoors in Machine Learning Models. FOCS 2022. arxiv.org/abs/2204.06974 Draguns, A., Gritsevskiy, A., Motwani, S.R., de Witt, C.S. (2024). Unelicitable backdoors via cryptographic transformer circuits. NeurIPS 2024. Wheeler, D.A. (2009). Fully Countering Trusting Trust through Diverse Double-Compiling. PhD dissertation, George Mason University. dwheeler.com/trusting-trust/ Cox, R. (2023). Running the "Reflections on Trusting Trust" Compiler. research.swtch.com/nih Hacker News discussion: news.ycombinator.com/item?id… Read Multiplex, METR: The New AI Measurement Gatekeepers and the Quiet Path to State Control of AI (September 13, 2026). readmultiplex.com/2026/09/13…