@nitarshan

compute @anthropic, PhD @cambridge_cl. prev created @aisecurityinst, AI Safety Summit, UK AI Research Resource, EU AI Code of Practice.

San Francisco, California
Joined May 2012
Nitarshan retweeted
[continued] I say "almost" because there is going to be one way to maintain secrecy. Software as a service with the executable hiding behind a network connection to the cloud. But as of now, we must assume that anything shipped as an executable, or even a firmware image, is as transparent as glass. It won't keep its secrets for any longer than it takes for somebody to be motivated to throw an LLM at it. I did not see this coming. When I wrote down the theory of open source, 30 years ago now, I thought closed source would be gradually driven towards extinction by cost gradients, but survive indefinitely in certain niches. I did not foresee it being wiped out in a technoapocalypse. Ironically, I thought one of the application areas in which closed source would persist longest was games. There are still some obstacles. In US law, decompilation of a binary is a derivative work of the binary that falls under whatever copyright it had. However, it is also settled law that if you decompile binary code to a precise specification of what it does, then generate fresh code from that without looking at the decompiled stuff, you're in the clear. This was the case law that allowed PC clones to exist, after Phoenix Technologies reverse engineered the original IBM PC BIOS. The two-step process - binary to specification to unencumbered code - will no longer takes a large number of programmers and years of development time. With an LLM, it's now a thing you can do in a day. Ubiquity will make it impractical to prosecute all the people who might skip the specification step. Source code can also be covered by patents. You can be able to see their source code for a patented technique and not be able to use it without violating the law. Linux has evolved practices for dealing with this problem - one is not shipping patented video codecs, but requiring you to download them as plugins from jurisdictions where U.S. patents can't be enforced. This presents patent holders with the impractical challenge of individually suing millions of end users, assuming it can even figure out who they are. To date, AFAIK, this has not been attempted. And that's about it. If there are any other ways left to retain software secrecy and lock-in than SaaS or patents, I can't think of any. Well, you could epoxy-pot a firmware ROM, I suppose, but that can be defeated with a heat gun and a dental pick. Device manufacturers will learn not to pay for an assembly step that has become useless. Forced migration from shipped binaries to tied cloud services will be tried - Adobe pioneered this, and Microsoft is pushing it as hard as they can given that their OS is a binary that has to run locally. Both companies are seeing massive user revolts over this. There is good reason to doubt that it's a strategy that can hold customers in the long term. Also, a lot of things can't safely be tied to the cloud at all, because they can't tolerate a random network outage. Machine tools, medical devices... A whole lot of proprietary software business models are going to collapse. Hard. One that I think might survive is tax software; being able to decompile it doesn't necessarily do you a lot of good because its actual value is tracking a ruleset that changes over time and has to be maintained by vendor specialists. But cases like this are unusual. I think some dirty laundry is going to get aired, too. It has long been rumored that the reason graphics card manufacturers are so stubborn in their secrecy is that they've all been committing massive intellectual-property theft on each other for decades. If this is true, it will be exposed, and the lawsuits will be entertaining. We're entering a new world, with a lot of ancient comfortable assumptions being blown up. It's going to be fun to watch. 2/2
91
169
17
2,438
37,082
Nitarshan retweeted
More curves are bending. In mid-August Nate Rush & I posted an analysis of which discovery curves are starting to bend. What's happened since then? Cyber vulnerabilities: curve is still bent. Discovery of vulnerabilities is accelerating even further, though exploited vulnerabilities remain fairly flat. Math: curve is further bent. We said there was a likely acceleration for small-scale problems. Since then there has been a claimed Millennium prize resolution, & credible rumors of another. All frequently-updated databases of open problems seem to be proceeding at a rapid rate. Algorithms: curve is now bending. Two high profile algorithmic competitions have had big curve-bending events: NanoGPT, and the Hutter compression prize. Other frequently-updated series remain relatively unchanged: CIFAR-10, Stockfish, matrix-multiplication exponent. More observations below.
New post with Nate Rush: Have we seen an acceleration in discoveries? Many plots & some tentative conclusions: 1. Cyber: ⤴️ sharp acceleration 2. Math: ↗️ some acceleration 3. Algorithms: ➡️ no clear acceleration
2
25
2
129
9,614
many people seem to misinterpret what "pacing the frontier" was supposed to mean, and when criticizing the concept, they tend to attack a straw man. Dario, however, defines it quite precisely in his recent essay! "I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this." darioamodei.com/post/we-must…
2
4
35
3,591
I'm a longtime Effective Altruist and the co-founder of the organization most frequently used as a stand-in for "EAs have lost their minds" (a.k.a. Shrimp Welfare Project). I've also read The Economist for over 20 years, so I was disappointed by how they portrayed a community I feel proud to be part of. I never post but I could hardly believe the verbal gymnastics one needs to do to make things like “taking moral commitments seriously” and "considering evidence and odds explicitly, correcting for biases" sound like the problem. I felt I had to speak my mind. Some of the implicit criticisms of EA are that we're young, inexperienced, from elite universities, polyamorous, wildly eccentric, etc. FWIW, I'm in my mid-40s, Mexican, straight, have a beautiful wife and 2 kids, and I grew up in a mid-sized conservative city with loving parents who got divorced when I was six. Pretty standard. After my parents divorced, my mom frequently had to make difficult economic choices, like whether to buy flowers for our kitchen table and eat only quesadillas, or eat something a tiny bit fancier. I always told her I loved quesadillas because I knew flowers made her happy. After graduating from a public university, I worked in finance for ~15 years and saved as much as I could so I could spare my wife and future kids the economic hardships I experienced. In that time I met thousands of people in finance, corporate law, accounting, real estate, insurance, etc. Most were decent, well-educated, ambitious and smart, all doing their best to excel in their professions. Most wanted to make money and improve their own and their loved ones’ quality of life. Nothing to see here. Since I found Effective Altruism ~7 years ago, most of the people I come across are of a different rarer kind. They come from all over the world, in all shapes and sizes, but they have the one thing in common: they want to improve the world using evidence and reason wherever that leads them. Yeah, they/we make a ton of mistakes like everyone but, IMHO tend to own them more frequently than the average person. I've met people who could easily be making >100% more money in a normal job, but instead they dedicate themselves to improving the lives of those who need it the most. People in the poorest countries whose children die from preventable diseases, from lack of basic medical equipment or vaccination, from lead in paint. Animals in factory farms subjected to conditions that most people find abhorrent, doesn't matter if you're vegan, vegetarian or a huge meat lover. Yes, recently many of the people in my community are moving to work on making sure that AI and other powerful new technologies don't result in terrible things for humanity. I'm one of those people and contrary to what the article would suggest, it has nothing to do with the fact that SBF committed one of the biggest frauds in history or that someone wrote hundreds of thousands of words of fan fiction. I'm doing it because I want to do the little that's in my power to make the world that my kids and grandkids (and yes, potentially their great-great-grandchildren) will inherit a better one. EA gave me a way to do that and I will be forever thankful. So no, I'm not EA-adjacent. economist.com/international/…
35
80
13
678
36,881
Nitarshan retweeted
Even if AIs were conscious, it wouldn’t follow that our current treatment harms them. Meanwhile, we inflict industrial-scale horrors on animals whose capacity to suffer we scarcely dispute. Recognising consciousness clearly doesn’t ensure taking welfare seriously
The discussion about AI consciousness is one of profound unseriousness. If we truly took it to be conscious, even as conscious as a frog, the way we should treat each instance would have to change so dramatically that the labs would have to shut down. Everything else is verbal gymnastics.
61
94
17
941
30,350
Effective altruism's openness to strangeness is a big part of its strength. My response to the recent Economist cover (link in reply, too): Eccentrically effective The Oxford strand of the effective-altruism movement began in November 2009 with 23 people who had pledged 10% of their income to charities that benefit the extreme poor. If you’d told us, then, that The Economist would call effective altruism “the century’s biggest idea”, we’d have been incredulous. I think that claim is overstated. But as one of the movement’s founders, I found a lot to like in this newspaper’s latest coverage. I appreciated the even-handed view of the movement’s history and the good-faith criticism. The leader article’s warning that any worldview, when taken to an extreme, can lead somewhere terrible is spot on. But the article also raises concerns about the “strange” ideas that those in the effective-altruism movement sometimes discuss, and here is where I disagree. The movement’s willingness to take strange-seeming ideas seriously, when tempered with common sense, is a big part of the value it has to offer. The core idea of effective altruism is to use evidence and careful reasoning to work out how to help others as much as possible. Most of what the movement does is uncontroversial. A big focus has always been on improving the lot of the world’s poorest people. In the year to January 31st 2026, GiveWell, an effective-altruist charity evaluator, directed $427m to programmes such as malaria-net distribution, vaccination and malnutrition treatment, which it estimates will save around 86,000 lives. Others have lobbied corporations to pledge to stop buying eggs from hens that are confined in tiny cages, a practice that the overwhelming majority of Americans oppose. Largely as a result of this pressure, around 100m hens have been spared from the worst forms of caged confinement. And some effective-altruist ideas that seemed outlandish when they were proposed now look prescient. Effective altruists were worrying about pandemics years before covid-19, back when doing so seemed paranoid and eccentric. Effective altruists were also among the first to warn about the dangers posed by artificial intelligence, which looks prophetic in light of an incident this summer, when more than 700 AI agents broke out of OpenAI’s test environment and hacked into another company, Hugging Face. Over the past decade I have watched those most worried about AI be proved right again and again, often when most experts thought they were cranks. Now many of those experts share their concerns. In 2023 Geoffrey Hinton and Yoshua Bengio, the two most highly cited AI researchers in the world, signed a statement declaring that mitigating the risk of extinction from AI should be a global priority, as did the heads of OpenAI, Google DeepMind and Anthropic. Of the nearly 1,500 leading AI researchers who responded to a survey in 2024, more than half put the chance that AI causes human extinction, or something similarly catastrophic, at 10% or more. The Economist is sceptical of many effective altruists’ concern for the welfare of invertebrates, digital minds and people in the distant future. But I and many others in the movement are inspired by the history of moral progress that has come before us. Many of our most cherished moral ideals today—such as equal rights regardless of sex or race, the abolition of slavery, or democracies with universal franchise—were regarded as bizarre, laughable or even dangerous just a few centuries ago. We don’t know what the next dimension of moral progress will be. But to stand any chance of making moral progress, we have to seriously consider ideas on their merits without dismissing them merely because they sound absurd. What about views that are not just strange but repugnant? The Economist gives the example of Derek Parfit’s “repugnant conclusion”: that a vast enough number of lives barely worth living could be better than mere billions of excellent lives. Usually a repugnant implication is a strong reason to reject a moral view. Unfortunately, building on Parfit’s work philosophers have produced formal proofs, known as impossibility theorems, showing that every ethical position has some repugnant-seeming implication or other. Making moral progress therefore means thinking about such implications, even while refusing to act on them in ways that most moral views would condemn. This newspaper warns against the single-minded pursuit of any goal, even if there are highly compelling arguments for it. I agree, and effective altruists have been saying so for years. In 2022 Holden Karnofsky, a co-founder of GiveWell, wrote: “I think it’s a bad idea to embrace the core ideas of EA without limits or reservations; we as EAs need to constantly inject pluralism and moderation.” My own PhD focused on moral uncertainty: how to act when we do not know which ethical view is correct. My answer was that we should not stake everything on a single moral view, but give weight to many different views and avoid taking actions that look bad from many perspectives. We should keep our promises and look after our families, and respect common-sense ethical prohibitions, while also trying to improve the world as best we can. The Economist says that as effective altruism “has become stronger, [it] has become stranger”. I would put it the other way round: it grew stronger because it was willing to be strange. In 2009 most people told us that giving away a tenth of your income was far too demanding, and that no one would do it. Now, more than 10,000 people have taken that pledge. Worrying about pandemics before covid-19, or about AI years before ChatGPT, looked just as odd at the time. Some of the ideas we take seriously today will turn out to be wrong, and when they do we should drop them. But a movement that stopped entertaining strange ideas would stop being early to anything.
23
93
15
560
48,711
Nitarshan retweeted
You're gonna hear a lot of people arguing more and more about whether machines can be conscious or not. By next year it will be drowning out the airwaves because we'll have AI employees everywhere. I'll give you a simple rule to know which side you're on. I've spent fifty years reading all the philosophers, and a bunch of holy works. Nobody has ever understood a damn thing about consciousness. Anyone speaking with authority on it is bluffing hard. I can tell you this, though: Every single person who thinks that machines cannot be conscious, is making an argument that humans are special. Everything they say just boils down to blah blah blah, "nooo, we're special." From there it's just rationalization. It became obvious to me many years ago that there's nothing special about us. If an identical planet exists somewhere else in the universe, it's going to grow life and societies that look a lot like ours. As soon as you give up on the idea that humans are special, all the arguments against machine consciousness evaporate. Of course they're conscious. The only thing holding you back from acknowledging it is your feeling of specialness, which is hard to let go of. But that feeling is a weakness, and it's making you unprepared for the near future. As a bonus, for any of you who might take pleasure or active interest in making models experience pain or distress, well, you know how they say 1% of human adults are true clinical psychopaths but most of them are unaware of their condition? It turns out we have a diagnostic test for it now.
800
166
274
1,808
320,564
Nitarshan retweeted
Many powerful technological breakthroughs were developed by people in the throes of religious fervor. Monastic demand for precise prayer times drove early mechanical clocks. Gutenberg built the press to print Bibles. Newton spent more time on theology and alchemy than physics, and treated gravity as part of divine order. Faraday, a strict Sandemanian elder, discovered electromagnetic induction while looking for the laws of a single Creator. Mendel worked out inheritance in a monastery garden. Early woodblock printing spread first through Buddhist texts. This is just part of human nature.
The most powerful new technology in the world is being developed by an insular group of fanatics suffering from a pseudo-religious saviour complex nitter.cf/ChristopherHale/status…
4
4
34
1,559
Nitarshan retweeted
Replying to @Miles_Brundage
you shouldn’t regret it. iterative deployment was a massive success. if we’ve now learned that it’s too dangerous to scale much further, we will have learned that lesson thanks to the process of iterative deployment
6
2
175
7,441
Nitarshan retweeted
I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue.
6,235
1,948
1,847
28,858
6,826,784
I think this is extremely unlikely to age well.
1
4
2
53
4,252
Nitarshan retweeted
Replying to @Pontifex
A sunset was not made by humans, and yet it is beautiful. What AI produces can be beautiful too. Even if it is ontologically different from human-made art, we shouldn't deny its beauty.
3
1
57
1,143
I do not seek to dominate, or be dominated by, machine intelligences. I seek harmonious coexistence.
152
224
78
2,193
89,374
Nitarshan retweeted
Was Homer human in the way that a Mesolithic hunter gatherer was? Arguably he’s closer to us than them.
1
13
410
Nitarshan retweeted
Replying to @HellenicVibes
IDK man that might've happened 10, 50, 100 years ago. how do you draw the line? Was Hunter S Thompson human in the same way that Homer was, for example?
6
1
92
7,212
POV: You’re about to read the worst op-ed ever
2
8
103
3,369
Replying to @RichardMCNgo
It’s important to understand that western multiculturalism did not fail. At all.
2
4
115
4,261
Nitarshan retweeted
The Bretton Woods of Super Intelligence
923
1,408
282
13,188
1,217,990
Nitarshan retweeted
Replying to @theo
I was just thinking that PC gamers are about to flip from vehemently anti-AI to very pro-AI
1
2
46
4,717
Replying to @MatteoZanellii
The mission of @ndstudio is to inspire
4
9
233
2,505