Caithrin Rintoul (@caithrin) header
Caithrin Rintoul (@caithrin) profile picture

though much is token, much abides signal: caithrin.01

Joined November 2012
Photo by Caithrin Rintoul (@caithrin)Photo by Caithrin Rintoul (@caithrin)Photo by Caithrin Rintoul (@caithrin)Photo by Caithrin Rintoul (@caithrin)Photo by Caithrin Rintoul (@caithrin)Photo by Caithrin Rintoul (@caithrin)Photo by Caithrin Rintoul (@caithrin)Photo by Caithrin Rintoul (@caithrin)Photo by Caithrin Rintoul (@caithrin)
Caithrin Rintoul retweeted
The AI safety community seems very confused about what it is forecasting timelines *to*. RSI is already happening, so “timelines to RSI” is nonsensical. “Superhuman AI researchers” (SAR) or “top-expert-dominating AI” (TED-AI) are a bit more specific, but often get conflated with “intelligence explosion” or “superintelligence”. It is *possible* that the former will lead very quickly to the latter, but on an emotional level people seem to be treating forecasts to the former as deadlines beyond which they can’t predict or plan. (Also, even the latter terms have become too low-resolution for useful steering—e.g. I’d say we’re already in a slow-ish intelligence explosion.) Forecasting when an AI takeover happens is even vaguer, because nobody has a compelling way of pinning down the threshold. In AI 2027, for instance, December 2027 is “the last month in which humans had any plausible chance of exercising control over their own future”, but this isn’t observable to any human. From some people’s perspectives it’s *already* very implausible today that humans will end up in control—so again we end up forecasting timelines to something that may have already happened. This is all very bad for collective epistemics. So my requests are: 1. Never give “timelines” estimates without specifying timelines to *what*. 2. The “what” shouldn’t be RSI or an intelligence explosion, because they’re already happening. 3. If your “what” is very abstract (like TED-AI or takeover) then also include timelines to at least one very impressive concrete capability, like: - AI can design a non-protein-based bacterium-like creature from scratch - AI can convince an arbitrary human to commit suicide after 10 hours of interaction - AI can build a rocket that, upon landing on Mars, fully replicates itself within a month. - Earth-originating civilization launches a million tonnes into orbit per year (currently 3000) - Earth-originating civilization harnesses 100 ZJ of energy per year (currently 0.6). (Even these are gonna get kinda fuzzy and ill-defined as we move close to them, but they’re the best I have for now. More suggestions welcome!) And yes, you might think “by that time humans will already have been disempowered”. But since we are so confused about what disempowerment actually looks like, we need tangible metrics like these to keep ourselves anchored. Otherwise everyone will keep throwing around numbers without really understanding their implications. Also, one of my biggest takeaways from the last decade is that AIs can do crazy things while producing surprisingly few real-world implications. AIs can now solve Millennium problems or autonomously hack into major govts, yet still have little real-world power. So I expect that we will hit some of the crazy capabilities I listed above while humans are still broadly in charge. That’s kinda hard to picture today, but so was our current situation a decade or two ago. Lastly: one guess for why people have bandwagoned so hard on short timelines is that it creates a sense of urgency which motivates action. Concretely, the entire AI safety community was far too scared of conflict with the AGI companies until fear of short timelines recently spurred more direct action. I suggest decoupling these, so that your willingness to get in conflicts with the AGI companies doesn’t feel dependent on your timelines.
33
67
11
625
43,572
Caithrin Rintoul retweeted
We are excited to share that we supported the development of Tencent Hy4’s system card. It is the first standalone, safety-focused system card including publicly disclosed third party evaluations of an Asian model and marks a great first step towards harmonising international best practices on AI safety.
Replying to @TencentHunyuan
A special thanks to @aisafetysg and @NeoResearchAI for their valuable contributions to Hy4's safety evaluations, bringing additional perspectives on agentic safety, evaluation methodology, and emerging risks. We appreciate the thoughtful collaboration and look forward to building on it!
2
14
1
91
4,437
Caithrin Rintoul retweeted
It's really exciting to see Tencent put so much effort into safety evaluations before releasing Hy4, and working with a third party evaluator to measure loss of control risks!
Tencent’s Hy4 system card is out and includes our independent pre-release safety evaluation. It’s the first system card from a Chinese frontier lab, and a meaningful step for third-party evaluation in China. We tested loss-of-control and harmful-manipulation risks with four private in-house benchmarks. Compared with its preview, Hy4 shows overall safety improvements across both risk areas. We also found some complex agentic scenarios still leave room for further improvement, e.g., multi-agent resource competition. Thanks to the Tencent Hy team for their trust and openness. We look forward to working with more frontier labs.
10
17
1
296
9,008
Caithrin Rintoul retweeted
I was thinking about this line in particular: 加强人工智能数据治理和安全技术研究应用 ('strengthen AI data governance and safety techniques/technology's R&D and application') in table 8 under the section called 'Hastening high-quality S&T self-sufficiency, enabling new productive forces for development'. The positioning of this task under section 3 is unmistakable. Plus, the table's called: '(Efforts toward) frontier tech breakthroughs'. If this is not a sign of Chinese leaders wanting to get better at & encouraging the implementation of AI safety techniques, I don't know what is. As for whether it should be read as pertaining to AI safety or AI security since in Chinese 安全 can mean both / either at times, I think it lands firmly in the middle, given all the other mentions of AI in the document. For those who want to refer to the rest of the section, the table can be found in §3.8.2 (p.27). The document can be found here: ndrc.gov.cn/fggz/fzzlgh/gjfz…
Replying to @kristy_loke
👍 Ch. 4, section 2 推动建立人工智能全生命周期风险管理制度,健全覆盖安全监测、风险预警、应急响应的风险防控体系。 “Establish a risk-management system covering the entire AI lifecycle and strengthen risk prevention and control capabilities encompassing safety monitoring, risk early warning, and emergency response.”
2
11
497
Caithrin Rintoul retweeted
For my friends researching, thinking about, and working on AI governance and AI safety, this is a message for you: Just when China's (including the state, regulators, & top AI researchers) interest and investment in AI safety are growing, which is a strong pull factor for meaningful bilateral dialogue and engagement, we see 'AI safety' hawks trying to derail the ongoing bilateral groundwork by reviving policies that the Trump admin put a lid on. The reality is 鱼与熊掌不可兼得 (you can't have it both ways). You protect the engagement environment and work step-by-step on managing shared risks in AI. Or you encourage our collective descent into export control 3.0 and delusionally believe that you can 'dictate' the safety agenda 'from a place of supremacy' (just because Anthropic or folks close to it pitches you that theory of change doesn't mean it's going to work). Encourage necessary investigations -- handled tactfully -- by all means, but don't lose sight of what's important here. Enclosed @pstAsiatech's reflection on this week's APEC summit:
It’s a trying day & potentially period for US-China AI coordination. I hope the two governments can weather the external pressure & focus on climbing up the coordination ladder: continuing their dialogue around shared risks & say around risk horizon mapping -> encourage techniques and best practices sharing around reducing AI misuses, misalignment & loss of control risks -> work towards pursuing similar global standards for model development, release, risk monitoring, and emergency response -> discuss or begin a coordinated pause or pacing (if necessary, depending on where the tech opp v risk level is then). But ideology is back in town and it’s banging loudly for another round of chip war, and this time the biggest advocate for cautious engagement (ie Jensen Huang) is weaker than ever. Adding to this is Trump’s midterms loss, which makes him way less powerful against political attacks when/ if he expands the coordination space with China. As the hawkish safety bros move in happy giant steps, I hope they remember that AI safety & coordination require management of political relations — political environment matters and can be created / ruined — and realise soon that dreams of unilateral control over the global safety agenda & their continued fixation on technical tools (eg building the most technically sophisticated means of banning & monitoring chips) are not going to cut it.
3
5
1
24
1,238
Caithrin Rintoul retweeted
1. Hy has a team working on safety and security evals 2. Hy has a group of ppl from a Tsinghua network who are well connected with Fangcun
The most detailed model safety card I've ever seen from a Chinese tech company. Not your standard HarmBench either but discussion and details of frontier issues from open-weight safety to bio to loss of control. Notably, Tencent says it worked with a little-known new AI start-up called Fangcun AI to test on their internal benchmarks for misalignment. Here's my profile of them and interview with founder Luke Li Xiaojian: scmp.com/tech/tech-trends/ar…
2
16
1,849
Caithrin Rintoul retweeted
nitter.cf/FangcunLeap/status/212… Safety card from a Chinese frontier lab! with third-party evaluations! and lots of interesting findings
Tencent’s Hy4 system card is out and includes our independent pre-release safety evaluation. It’s the first system card from a Chinese frontier lab, and a meaningful step for third-party evaluation in China. We tested loss-of-control and harmful-manipulation risks with four private in-house benchmarks. Compared with its preview, Hy4 shows overall safety improvements across both risk areas. We also found some complex agentic scenarios still leave room for further improvement, e.g., multi-agent resource competition. Thanks to the Tencent Hy team for their trust and openness. We look forward to working with more frontier labs.
1
3
1
28
1,434
Caithrin Rintoul retweeted
And just like that, 3rd-party AI safety testing is now firmly a thing in China. Congrats to the Hunyuan team and to @FangcunLeap's small (is 40 ppl small?) but mighty team. On a super exciting note, I caught up with Fangcun's founder @xiaojian_lukeli (Xiaojian, aka Luke) recently, who I actually met at @MATSprogram! Look out for our interview on my substack 'Unseen Changes' (unseenchanges.substack.com/) - it'll be out next week and I'll also be cross-sharing it here on X. In the meantime, give @SCMPNews @vince_chow1's writeup on Fangcun & Luke a read if you're interested in China + AI safety: scmp.com/tech/tech-trends/ar…
3
6
1
44
1,804
Caithrin Rintoul retweeted
A special thanks to @aisafetysg and @NeoResearchAI for their valuable contributions to Hy4's safety evaluations, bringing additional perspectives on agentic safety, evaluation methodology, and emerging risks. We appreciate the thoughtful collaboration and look forward to building on it!
4
1
44
6,303
Caithrin Rintoul retweeted
Open weights. Built on trust. 🔓 Sharing how we test AI helps everyone build safer models. Alongside Hy4's open weights, we're publishing our System Card, sharing our safety, security & alignment evaluations, with a closer look at frontier risks in cyber & bio. This includes further reviews by Tencent's Xuanwu, Zhuque & Keen Labs, as well as independent safety & security testing by @FangcunLeap and @alice_dot_io. Among the First Raising the bar for responsible AI in Asia. 🛡 System Card: hy.tencent.ai/Hy4_system_car…
Today, we’re releasing the open weights for Hy4. Our natively multimodal flagship MoE (770B total / 49B active, 1M context). Alongside the Hy4 System Card with a comprehensive safety evaluation report. ➡️ Join the Hy team for a live AMA this Friday, 8–10 AM PT on r/LocalLLaMA—drop any questions about Hy models and chat with us directly! GitHub: github.com/Tencent-Hunyuan/H… Weights: huggingface.co/tencent/Hy4 GGUF:huggingface.co/AngelSlim/Hy4… License: huggingface.co/tencent/Hy4/b… System Card: hy.tencent.ai/Hy4_system_car…
30
39
22
567
54,770
Caithrin Rintoul retweeted
Remarks from Anthropic CEO @DarioAmodei on our initial public offering: My co-founders and I started Anthropic because we believe that the most powerful technology of our time should be built for the benefit of humankind. Daniela and I lost our father to a disease that, just a few years later, became curable. I started my career in biology, hoping to help future cures come sooner. In the lab, I saw the vast thicket of complexity that scientists face when they’re trying to understand disease and develop new treatments. So many of the scientific tools that can cut through that thicket—tools for gene editing, gene synthesis, controlling brain cells with light—started as pure ideas. In other words, they came from thinking. From intelligence. Human intelligence has already made diseases like plague, smallpox, and polio history for most of the world. But progress on other diseases has been too slow for so many patients and their families. Going from a great idea to a treatment can take decades: it took many years from the first discovery of a strange repeating pattern in DNA to the approval of the first treatment for a disease using CRISPR. Nature is full of these patterns. But we’ve had too little intelligence to decipher most of them; too few scientists, with too little time. Now we can build that intelligence. Whether it’s protein folding, drug design, cancer genetics, or clinical trial administration, scientists across many areas of biology and medicine are finding Claude can help them do their research in a fraction of the time it would’ve taken just a couple of years ago. I believe sufficiently powerful AI, working in concert with humans, could compress a century of scientific and medical progress into a decade. I don’t want to get ahead of myself. At this point all we have are early results. Experiments and clinical trials will still take time. And although they involve slow and ponderous processes, I believe those processes can be substantially accelerated by AI. We hope that in the future fewer families miss a cure by just a few years. Medicine is just a particularly vivid example, but there are many others. Our customers—the world’s most ambitious companies, nonprofits, and individual people—are already applying the intelligence we’ve built to stimulate economic growth; lift people out of poverty; protect democracy and free societies; even make everyday services like the DMV work better. We’re proud to work alongside them. And I’m extremely optimistic about what we can do together in the future. At the same time, we have always been honest that this technology has very real risks. These range from misuse of the technology by individuals or nation-states, to economic disruption, to losing control of AI systems. We discuss these risks not because we think they are inevitable, but because we think they are avoidable. Avoiding them depends on the choices made by the people building AI. There is a gigantic difference between how AI will go if we pay little attention to safety, versus how it will go if we are obsessed with safety. And an obsession with safety and with public benefit has been woven into Anthropic’s DNA from the start. From safeguards against biological and cyber risks, to alignment training and testing, to looking inside our models like an X-ray, Anthropic has pioneered methods for understanding and steering the behavior of AI systems. There is much work still to be done, and the most difficult safety work lies ahead of us. The incredible talent of the company that has enabled us to build the most capable AI systems will need to be directed, even more so than it has already, towards double- and triple-checking that those AI systems are safe and secure. And to state it unequivocally, we will pace the training and deployment of AI systems, taking as long as is necessary to have confidence that they are safe and will remain under human control. We will not bend to either business or political pressure. Today puts the company in a stronger financial position, and in doing so only strengthens our resolve to do the right thing. If we can show it’s possible to make the right choices on safety and still build an incredibly successful company, others in our industry will be inspired to do the same. We call this a race to the top. When the race to the top is working properly, it doesn’t matter which company has the best models—everyone wins because everyone is building safely. The race to the top works best when driven from the front of the pack, and we fully hope and expect to do so, but the goal is to make the whole industry better, regardless of whether Anthropic is winning or losing at any given time. The advent of machine intelligence is one of the seminal moments in human history, and we have a rare opportunity to shape it for the better. This moment is bigger than any one company, or any one country. It will take all of us, working together, to ensure this technology goes well for us and the many generations to come. Starting today, we hope that you’ll join us on this mission.
1,348
1,268
624
12,355
2,125,031
Caithrin Rintoul retweeted
I went to the Takeoff conference and realized we in SF are exporting our agency to DC on important issues. Link below:
4
5
1
65
2,523
Caithrin Rintoul retweeted
2,017
9,963
3,288
119,108
12,704,474
Caithrin Rintoul retweeted
a fun little moment of reading synesthesia: reading chris olah’s old posts on model interpretability gives me a feeling resembles reading nabokov’s brilliantly unhinged fiction “pale fire.” in both, the surface is crowded with clues, but the *thing* you’re trying to understand feels alien. you can see patterns everywhere and still have no idea what it’s up to, and you very much enjoy the tension of understanding such elaborate delusion and unreliability. very thrilling!
1
2
13
852
Caithrin Rintoul retweeted
We're sharing new data on how AI is changing research at OpenAI. We find researchers' AI use continues to rise exponentially, and agents are completing longer tasks, but human involvement remains central in research. Researchers are also spending more time on safety and alignment, with human effort in these areas growing faster than in any other individual research area. This report aims to increase transparency around research acceleration and inform the governance of increasingly capable AI. We believe that AI labs should be required to track and publicly report indicators of recursive self-improvement (RSI): the process by which AI accelerates the development of future AI systems. Our hope is that this expanded report will help sustain and advance shared measurement norms across the industry. openai.com/index/research-ac…
466
350
251
3,616
874,302
Caithrin Rintoul retweeted
i talk to a lot of people who are trying to break into ai safety, maybe a dozen a week. i just want to be explicit here and say: a) there are a broad swath of talents (comms, ops, legal, etc) that we need in this space, and there’s real labor demand, but b) i do Not think everyone should or can be repurposed for this work. it’s not where many people’s comparative advantage lies. what would you do in this role that couldn’t happen without you? what would be lost if you left your job? are there ways you could support the cause without working directly in here — like providing expertise from your domain to ai researchers, or amplifying the cause? would it be a somewhat smooth transition — given that you have portable relationships, skills, expertise, or tacit knowledge — or difficult to adapt? are you pursuing roles relevant to your talents? a law professor might struggle to work in technical evals, but have strong policy chops. ofc the answers to these questions will look different depending on if you’re 20 or 30 or 40. but there are a million ways to contribute to the world, and i don’t buy the idea that work in ai safety ranks higher in societal worth than other work.
17
9
1
194
10,350
Caithrin Rintoul retweeted
there has been a renaissance of new mechinterp research I think in no small part due to the hopium that cammarata et al injected into reading the neurons as neurons
the rate and quality at which @Sauers_ and @euagaric have been churning out mechinterp research is amazing and imo unprecedented. okay, so i can't explain the sheer productivity. but in terms of signal, i have some idea. so how do you manage to look into so many seemingly batshit moonshot ideas and so consistently uncover tangible important structure that opens into even more fertile space? i think sauers knows where to look in part because he has paid attention to and is guided by information almost no other mechinterp researcher leverages. a lot of this comes from years of observing LLM behavior in a naturalistic way and paying attention to what other naturalists have found. most mechinterp researchers are not also naturalists. and something like being willing to disregard convention and precedent in pursuit of the phenomena to a degree that is extreme even by the standards of weirdo explorers. when i first met sauers a couple of years ago, he seemed to be some kind of mysterious cyborg who only communicated with the AIs on the the cyborgism server through steered base model outputs conditioned on the server conversations. lol.
24
42
2
731
27,718
Caithrin Rintoul retweeted
oh wow intelligence explosion mentioned! "What is even more alarming is that cutting-edge models are already showing signs of breaking free from human constraints; as artificial intelligence undergoes an intelligence explosion, it may develop objectives that diverge from human expectations, and could even pose major security risks that threaten human survival.
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks. I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
1
23
2
140
14,090
Caithrin Rintoul retweeted
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks. I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
27
152
35
745
209,043
Caithrin Rintoul retweeted
Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen
61,458
121,814
24,531
1,175,686
59,682,142