Quanton, the Spark/SQL engine from Onehouse, vs OSS Spark at Uber's production scale:
- 5.5x faster on 60B-record inserts
- 4.7x faster on 10B-record updates
- Beat OSS Spark on 8x fewer cores
#ApacheSpark #DataEngineering #Quanton
More than 200,000 CPU cores power Uber’s data lake ingestion. At that scale, every efficiency gain counts.
Making better use of that compute means examining how Spark executes queries, moves data, and writes to growing tables. Accelerating one stage can leave other bottlenecks untouched.
Uber engineers are improving production workloads with native vectorized execution and evaluating new ways to accelerate write-heavy pipelines.
Explore the engineering choices behind those efforts, where the benchmarks showed gains, and what remains to be optimized.
1
39
For AI data I think updates and deletes matter more than appends.
hudi-rs 0.6.0 does them from Python in under a second, no JVM, using a record-level index and Merge-on-Read.
Upserts 1.7-12x and deletes 4.8-24x faster than both delta-rs and PyIceberg on 4 of 5 workloads 👇
1
3
159
Blog: hudi.apache.org/blog/2026/11…
Reproduce it: hudi.apache.org/docs/python-…
13
Apache hudi-rs 0.6.0 is out 🦀 Upsert and delete rows in lakehouse tables straight from Python and Rust, no JVM, no Spark. pip install hudi. Sub-second upserts, updates and deletes on a single node over millions of rows, Python reads 3-6x faster, point lookups up to 24x. Blog 👇
hudi.apache.org/blog/2026/11…
Release notes: github.com/apache/hudi-rs/re…
Get started: hudi.apache.org/docs/python-…
pip install hudi==0.6.0 | cargo add [email protected]
19
🚀 Apache hudi-rs 0.5.0 is officially out! We’ve packed massive performance gains and core ecosystem updates into this release. 🦀💧
Check out the full release notes: github.com/apache/hudi-rs/re…
#ApacheHudi #RustLang #DataEngineering #DataFusion #OpenSource
1
3
95
🔥 Highlights:
• Blazing fast: TPC-H 100GB benchmark ~3.6x faster than Spark! ⚡
• Table Version 9 & 1.x format compatibility.
• New Streaming API, plus native HFile & Lance reader support.
• DataFusion 54 upgrade & direct Python API provider bindings. 🐍
1
30
🙌 Huge thanks to the community who made this happen, and a special welcome to all the developers who made their FIRST contribution to the repo in this release: @brayanjuls, @linliu-code, @crrow, @slfan1989, @sushiljacksparrow, @codephage2020, and @yunchipang!
19
Y Ethan Guo retweeted
Async indexing isn't a scheduling optimization.
It's an architectural consequence.
Stable mutation containers.
Timeline-mediated visibility.
Deferred reconciliation.
Those three primitives let Apache Hudi introduce new metadata indexes without interrupting ingestion.
Here's the deep dive into the architecture behind RFC-45.
#ApacheHudi #DataLakehouse #DataEngineering #DistributedSystems #Systemdesign
Y Ethan Guo retweeted
JD.com built a 125+ petabyte retail data lake on Apache Hudi.
Record-level mutations on commerce data at planetary scale ↓
Y Ethan Guo retweeted
Our Data + AI Summit booth number is 403.
Yes, the HTTP error code for "Forbidden". 🏴☠️🦜
But you've seen the movies. The treasure is always in the forbidden place. The sealed cave. The remote island. The door someone told you to stay away from. That's where they keep the good stuff.
Booth 403 is that door. Behind it: Quanton, our new Spark engine — ~3x the Spark for the same spend. If you're paying a fortune and still watching progress bars crawl, this is the chest you've been digging for.
Come plunder. See what else is out there. And throw your name in for some Nintendo Switch 2s while you're at it. 🎮
X marks the spot. Booth 403.
#DataAISummit #apachespark #databricks
The hidden tax of building production AI is fragmented storage layers. Lakehouses historically lacked support for embeddings and raw binaries. @apachehudi 1.2.0 solves this problem by unifying multimodal data for Analytics and AI into a single transactional lakehouse table.
🤖 Made with AI
2
35
Read the full announcement blog: hudi.apache.org/blog/2026/06…
21
Excited to moderate the "Open by Design" panel at #OpenXData2026! 🚀
We’re bringing together leaders from across the @apachehudi , @ApacheIceberg , @prestodb , and @ApacheFlink ecosystems to talk about why community is the true foundation of open data.
I am thrilled to be taking the stage with Yufei Gu from @Snowflake at #OpenXData 2026! 🚀 We’re diving into "Polaris Meets Apache Hudi": discussing how we’re unifying lakehouse metadata and governance across table formats without compromise.
📅 April 29 | 12:50 PM PST 🔗 Register here: openxdata.ai/
#ApacheHudi @apachehudi #ApachePolaris @apachepolaris #DataEngineering #OpenSource
1
39
Y Ethan Guo retweeted
Most people still think “interactive PySpark” means locking yourself into a premium Spark vendor, their catalog, and their pricing model.
⛓️💥 Today, that assumption breaks.
👉 onehouse.ai/blog/introducing…
We’re bringing Python Notebooks to Onehouse — but the real story isn’t “notebooks.” It’s what happens when notebooks run on an "open, vendor-neutral, BYOC data lakehouse" with 2–4× better price-performance.
A few things that actually matter:
1️⃣ Bring any catalog. Hive, Glue, Unity — your choice. The engine shouldn’t dictate your metadata or governance.
2️⃣ Keep your data (and cost savings) in your own cloud account. No ingest tax, no data gravity games, no discount gimmicks.
3️⃣ Quanton under the hood. Makes Spark/SQL workloads 2–4× more efficient with native support for Apache Hudi and Apache Iceberg.
4️⃣ Works with the workloads you already have. Any Spark/SQL job, now cheaper and faster — without rewriting everything around a single vendor.
Interactive PySpark is just the surface area. Open architecture + BYOC is the actual shift towards real cost/perf gains
Y Ethan Guo retweeted
🎉 Apache Hudi 1.1 has landed—and it's a banger! 🚀
This release is packed with innovations that will transform how you build data lakehouses:
🎯 Pluggable Table Format Framework for multi-format support (Hudi, Iceberg, Delta Lake)
⚡ 15x faster clustering with Parquet binary copy—yes, you read that right!
⚡ 2-3x higher throughput for Flink writers!
🚀 Spark 4.0 & Flink 2.0 support —harness the latest engine capabilities
Read the blog: hudi.apache.org/blog/2025/11…
#ApacheHudi #ApacheSpark #ApacheFlink













