Mark Kirschner

Tomorrow, @Tesla will turn on a massive and very expensive 10,000 unit NVIDIA H100 GPU cluster to help it train FSD. But that got me wondering, what is the difference between these new H100 GPUs and the older A100 graphics processing units (GPUs) Tesla has been using for the last couple years? I briefly break it down below. NVIDIA A100: This GPU launched in 3 years ago in late-2020. It introduced a 20x performance improvement over the previous generation. The A100 is designed for high-performance computing and artificial intelligence (AI) workloads: • 6,912 CUDA cores • 432 tensor cores • 40 GB or 80 GB of high-bandwidth memory (HBM2) NVIDIA H100: This ~$40k GPU launched in late 2022. Up to 30x faster than A100, and is up to 9x faster for AI training. The H100 is designed for graphics-intensive workloads such as video training (FSD videos), and is easy to scale up: • 18,432 CUDA cores • 640 tensor cores • 80 streaming multiprocessors (SMs) • Higher energy usage than A100 With the H100, high performance computing is over 5x faster compared to A100. These new H100 GPUs will enable Tesla to train FSD faster and better than ever, but NVIDIA can't keep up with GPU demand. As a result, Tesla is spending $1 billion+ to build its own supercomputer named Dojo. It uses the company's hyper optimized custom designed chip. Tesla is MUCH more than just a car company. This supercomputer will also train Tesla's fleet of vehicles and process data from them. @elonmusk said last month: “Frankly...if they (NVIDIA) could deliver us enough GPUs, we might not need Dojo.” Tesla is bringing online its NVIDIA H100 GPU cluster at the same time it's activating Dojo. This will dramatically increase Tesla's compute capabilities to a level that no other automaker could dream of right now. Take a look below at Tesla's internal forecast for the compute power of Dojo. Brace yourselves, everyone. Tesla's FSD V12 end to end training is compute bottlenecked, but the company is taking active measures to ensure that it won't be in the future. According to Elon, Tesla will spend over $2B in 2023 alone on training compute, and will do so again in 2024. Buckle up everyone, the acceleration of progress is about to get nutty!