nathan

There's a crazy reason behind how I understood the true importance of fully decentralizing compute. The story needs telling, so here goes. Back in 2015, I'd long shifted to being part of the early crypto scene, but maintained Fight My Monster, a venture-backed MMO game & social network I previously founded that had 3M users, which was self-sustaining financially. I'd found a long-term home working on decentralized compute, but couldn't let down the community that played the game. Fight My Monster was built in 2010-2012, using what was a completely novel architecture at the time that enabled it to scale (i.e. add massive amounts of users) at low cost, and with minimal administration. Essentially it comprised three core backend components. The first, was simple web hosting, which served the frontend game code and assets into its users' web browsers (I used a popular content distribution network). Just like the web-serving, the two other backend components had to be "horizontally scalable," which means that to support more users, and the associated compute and data, all I had to do was add more hardware, without any need to rewrite the backend software code, or perform excessive systems administration. This enabled me to scale fast in the early days, with minimal staff and financing. The second component, was a horizontally scalable game server framework called "Starburst", which was a work of love I created myself. This was built on top of an open source server that supported RTMP, an Adobe network protocol that allowed Flash media (remember that!!) assets running inside web browsers to directly talk to it. Starburst hosted game/social network logic and routed huge numbers of events being generated by the game to users in near realtime. The third component was a horizontally scalable NoSQL database called Cassandra (if you follow databases, you'll know that a modern derivative called ScyllaDB is currently doing well serving realtime AI needs). On my team, I had the only Cassandra core committer outside the company that built it. Further, I was the first serious party using the Cassandra beta in anger, and it got corrupted at 800,000 users, which was followed by a very testy 24 hours recovering the data, and bringing it back online (one moment I was going to the moon, the next moment I thought things had gone irreversibly wrong, but luckily things worked out). What was cool about Cassandra was that you could literally just add nodes (i.e. server machines with big hard drives installed) to horizontally scale its storage capacity, read throughput, and most importantly, its write throughput, which is very rare thing, and in my application absolutely crucial, because Fight My Monster generated a simply vast amount of data. Of course, another very important aspect of Cassandra was that it was fault tolerant, and I used a replication factor of 5, which meant everyone's data was very safe (or so I thought...). A replication factor of 5 meant I only needed nodes 3 online, for any specific "quorum of data," for it to be writable. Unlike a blockchain, Cassandra wasn't Byzantine Fault Tolerant, which meant that I had to manage the security of my infrastructure carefully to make sure hackers or malware didn't get in. But back then, hackers weren't interested in an MMO game/social network where most of the users were kids, and this wasn't supposed to be a general platform for others to build on. My main technical work pressure was just that the nodes hosting the database were under enormous strain owing to the amount of data being generated, and would therefore regularly blow up! This meant that I would have to login to my infra every so often, decommission broken nodes, and add new ones, which was easy enough, because the fault tolerance meant the job didn't have to be done immediately, and the "bare metal" machines involved could be orchestrated through my cloud provider... Fight My Monster used a setup years ahead of its time, and it allowed me to scale up with ease. Moreover, the fault tolerance, and the simple backend architecture with only 3 core system components, meant minimal administration and stress was involved in keeping it running. So what was the important lesson, which plays an important role in my thinking, and in ICP, to this day? You must surely have sensed that something crazy happened, and now I will share it publicly for the first time. I was traveling in Asia, as part of the "traveling circus" associated with the early core Ethereum community, visiting related parties (people often forget the importance of Asia, and China particularly, in helping support the early crypto scene) and attending Ethereum-related events, where I mainly talked about complex Computer Science techniques I had developed that I wanted to be used to make networks like Bitcoin and Ethereum scale and perform better. I don't want to drone on about "what is used to be like in crypto," but I will say that it was a magical time, and I owe a lot to early people for some amazing memories. It was also engaging, and I wasn't checking my email. Consequently, what I did not see on my travels, were email after email warning me of the fact that the credit card that was paying for Fight My Monster's Cassandra nodes had expired.... So what did the cloud operator do when I didn't respond within a few weeks (it wasn't many, but I forget the exact number). Well, they simply deleted all the nodes!!! It's hard to describe how I felt when I got back home to Palo Alto in California, and discovered what had happened. It was a mixture of shock and horror, a kind of hollow feeling in my stomach, adrenaline surges, and unfolding dread, as I worked as fast as I could to fix it, only to confirm that what had happened was irreversible. People who've been hacked in crypto will know the feeling. That was it. Fight My Monster was dead. Some part of the community even created a change dot org website pleading for it to be brought back online. Despite the loss of what it might have been worth, and the guilt, in some ways it was liberating, as I no longer had responsibility to maintain it, which probably would have done ad infinitum for the community. Plus it became a fond memory for a generation of kids and some adults that played it, before the game had a chance to grow too old and tired, and it allowed me to focus on what had become my obsession — decentralized compute networks. The lesson however was very clear to me. You can create decentralized compute infrastructure that can scale, and is fault tolerant, but if you have a single point of failure, anywhere, at any moment, it can be toast. That's why I'm always concerned when I hear about blockchains that depend on administrator keys held by the core devs, or blockchains that go down, or almost go down, when a huge chunk of their nodes are shutdown by a cloud running them, or because the cloud running them has technical problems. This problem of great fragility is vastly more prevalent than you might imagine in our industry. The learnings from the Fight My Monster disaster informed how the Internet Computer was designed. It's not only the number of nodes that matters, there are more important things. If nodes are anonymous, the number of independent parties running the bulk of them might be far less than you might think. Further, while large in number, they often turn out to be running on a small number of clouds— and even if they are owned and operated by a few different parties, the cloud might suddenly decide it doesn't want them anymore, and shut them down overnight, as Hetzner once did to a famous blockchain. What matters in practice is the number of independent parties running the nodes, and the physical and jurisdictional independence of the nodes. This simple insight is that was behind "deterministic decentralization," which will appear in new form inside the forthcoming "ICP cloud engines" extension of the Internet Computer. ICP cloud engines allow enterprises (typically) to create what is effectively an Internet Computer subnet whose configuration they themselves control, by choosing nodes to combine from a pool that acts as a marketplace. Currently there are more than 1.5k nodes waiting, and the number is growing. An engine provides a serverless cloud platform with remarkable properties. It is the ideal place for AI-built apps and services, because the has been stack designed to allow AI to generate advanced AIware software with ease, which AIware is secure and resilient. Apps are tamperproof (immune to infrastructure hacks), always-on (and thus incredibly resilient), optionally autonomous (which means there is no backdoor), digital assets native, and many other things. Engines also run e.g. the Open SaaS suite of AIware apps that can be used to run enterprises end-to-end. Cloud engines make installing and maintaining complex apps like Open CRM or Open Email as simple as installing and maintaining phone apps. Nodes are hot-swappable. That means you can change the compute replication factor, swap out the nodes for different ones, without interrupting what the cloud engine hosts, providing freedom from compute vendor lock-in, and, if you want, you can use a new class of nodes on hyperscalers, as well as traditional Internet Computer nodes, so it will function as a general compute marketplace. The cloud engine framework guides people to combine nodes that have independence, but provides flexibility that shared Internet Computer hosting doesn't have. For example, an enterprise in Germany might choose nodes that reside only in the EU, to meet GDPR requirements. However, the cloud engine console, which is autonomous, and served from the Internet Computer network, and updated by the NNS (the Network Nervous System, the world's most advanced DAO), guides users and warns them heavily if they combine nodes from the same node provider, or which are based in the same data center, say, or run by the same cloud operator. Thus, even with ICP cloud engines, the lessons of Fight My Monster live on. People will wonder why the framework pushes them so strongly to combine independent nodes, but there is a good reason, which is just as relevant for the wider blockchain community today, as it was in 2015. Looking forward to sharing cloud engines soon.