Databricks Japan🧱

The reason Databricks is the largest open source company in history is both structural and execution focused. Ben gives a great overview but as an open source founder I’ll try my hand as well. Databricks is at its core a compute company. Their project Spark beat out all the other distributed data processing architectures like Hadoop and Cassandra to emerge victorious in the big data wars of the 2010s. Partially this was because the other architectures were flawed and partially because they invested more in fixing them through hiring incredible engineers. This company did come out of a Berkeley PhD lab. Still they struggled to make money. So in the mid 2010s they began making their next investment in a proprietary execution engine for Spark they called Photon. As if building the leading distributing data processing library wasn’t hard enough they also shipped a native C++ vectorized query engine and continued both the public Spark roadmap and an internal proprietary roadmap as well. It’s worth pointing out that this company is a decent example of the a16z thesis that you can kingmake with capital. Ben dropped a $10m check on a company that only asked for $250k, which is how the company had the war chest to make these investments. If that’s all they had done, my guess is they’d be a $20B or $30B company today. I’ve made the point before that open source infrastructure expands forever. That said there was a big second act. Circa 2018 they pioneered lakehouse architectures. The way to think about it that the traditional way you provision databases is you reserve some bucket on specialized compute and storage for them on your infrastructure. This made sense in a pre-cloud world where you had separate special-purpose servers, but less so in a world where you have scalable services like S3, EC2 etc. So instead of reserving expensive DB compute and storage you reserve cheap S3 storage and compute that can spin up and down based on variable usage. And now there’s a third act, namely building a cloud with AI primitives. The new gateway is a big piece of that but last month they also released a backend as a service with the primitives for building agents. So again, why is it such a big company? Again, I think this is both structural and execution oriented. The structural reasons: they started in an extremely lucrative market with a barrier to entry (it is very difficult to create popular open source projects in valuable markets!), that expands forever. Second, they are a general purpose orchestration framework. What I mean here is that every open source / devtools market is split between general purpose orchestration and specialized single purpose utilities. Think auth (Clerk) or email (Resend) for web apps (Next.js), or integrations (Composio) and sandboxes for agents (Mastra). While it is simpler from a product execution standpoint to build a single purpose utility, it is usually more valuable to build the general purpose orchestration framework and Databricks bears that out when you compare them to single purpose utilities doing things like ingestion and transformation (dbt, Airbyte). Lastly as an orchestration framework / compute company, unlike database companies they are not limited to a particular workload (Supabase etc for transactional data, Clickhouse etc for columnar data), and they are in the core infra layer as opposed to being stuck on the client (Vercel/Next.js). It’s the $10B vs $200b difference The execution reasons: they are technical geniuses who really want to win. I’ve spent the last decade building open source companies (Gatsby and now Mastra) and the typical open source founders are brilliant tinkerers and, in the best case, it takes a while to develop a ruthlessly efficient competitive commercial DNA. With Databricks they had both sets of DNA within the founding team — incredibly rare. Congrats to @alighodsi and team on everything they’ve built so far!

🔓「オープンフォーマット対応」だけで、本当にオープンと言えますか？ 真にオープンなカタログを見極める3つの条件と、Unity Catalogの取り組みをご紹介します。 ▶︎databricks.com/jp/blog/what-… ✅ データは「お客様のもの」 ・テーブルはプロバイダー所有ではなく、お客様自身のクラウドストレージに置くことが重要です。Unity Catalogでは、マネージドテーブルの99％がお客様所有のストレージ上にあります。 ⚙️ 自分のストレージでも機能はフル活用 ・「マネージド」は所有ではなく、保守を担うという意味です。自動最適化や不要ファイルの削除、削除したテーブルの最大30日間の復元などを利用できます。 ・Zeptoでは、年間100万ドル以上のストレージコスト削減を実現しています。 🔄 コピーせずにカタログを移行 ・DatabricksはApache Iceberg RESTカタログ仕様にUNREGISTERを寄贈しました。REGISTERとの組み合わせで、データをコピー・移動せずにテーブルの登録先を変更できます。 ・現在はプライベートプレビューです。ご希望の方は担当アカウントチームへご相談ください。 #Databricks #UnityCatalog #レイクハウス