Apache Hudi retweeted
At scale, table metadata explodes, as Google's VLDB paper "Big Metadata" shows.
Hudi's metadata table treats it like big data ๐: self-managed, indexed
Key bits:
- Partition/file listings ๐
- Column stats ๐
- Record indexes ๐
This ensures efficient reads/writes as things grow ๐.
Apache Hudi retweeted
2/
Apache Hudi continues to operate at trillion-record scale inside one of the most demanding data platforms in the industry.
Not experiments. Not edge cases. Production systems moving hundreds of petabytes.
At Uber, Hudi is powering:
โข high-volume ingestion
โข continuous updates & merges
โข replication
โข incremental processing
Apache Hudi retweeted
[BLOG] Just published a blog on a problem that trips up many teams: retry storms from concurrent writers on data lakehouses
Streaming ingestion + long-running maintenance jobs = OCC fails you. Long jobs lose every commit race and retry indefinitely
@apachehudi NBCC solves this by eliminating conflicts by designโwriters append in parallel, completion time determines order. No retries needed.
Deep dive ๐
hudi.apache.org/blog/2025/12โฆ
#ApacheHudi #DataLakehouse #DataEngineering
Apache Hudi retweeted
The pace of innovation in the @apachehudi ecosystem is accelerating fast. If you want to keep up with these architectural shifts and production lessons, the monthly newsletter is the best way to stay plugged in.
๐๏ธ Read + subscribe:
hudinewsletter.substack.com/โฆ
Apache Hudi retweeted
New release: @apachehudi 1.1 is out!
The standout innovation: Pluggable Table Format Framework decouples Hudiโs storage engine from the format, unlocking full read compatibility with Iceberg and Delta Lake.
Itโs a massive unlock: Hudiโs ingestion performance with total format interoperability. Read my full write-up here: hudi.apache.org/blog/2025/11โฆ
#ApacheHudi #DataLakehouse
Apache Hudi retweeted
๐ Apache Hudi 1.1 has landedโand it's a banger! ๐
This release is packed with innovations that will transform how you build data lakehouses:
๐ฏ Pluggable Table Format Framework for multi-format support (Hudi, Iceberg, Delta Lake)
โก 15x faster clustering with Parquet binary copyโyes, you read that right!
โก 2-3x higher throughput for Flink writers!
๐ Spark 4.0 & Flink 2.0 support โharness the latest engine capabilities
Read the blog: hudi.apache.org/blog/2025/11โฆ
#ApacheHudi #ApacheSpark #ApacheFlink
Apache Hudi retweeted
๐ The Definitive Guide of @apachehudi has arrived!
This was a collaborative effort with my co-authors Prashant, @SudhaSakthee , and @RebeccaBilbro , bringing together years of collective knowledge from the Apache Hudi community โ from fundamental concepts to production patterns for building data lakehouses.
Grateful to @byte_array for writing the foreword, and to all the tech reviewers, editors, and those who provided praise and support for the book.
Thanks to @OReillyMedia and @Onehousehq for making this happen, and to everyone in the Hudi community who's shaped this technology.
For anyone building data lakehouses or exploring Hudi, hope this guide proves useful. ๐
#ApacheHudi #DataEngineering #DataLakehouse
Apache Hudi retweeted
Speaking at Open Source Data Summit (Nov 13) on building streaming-first lakehouses with @apachehudi ๐ค
We'll dive into:
โข Merge-on-Read tables
โข NBCC for conflict-free writes
โข LSM Timeline
โข Solving the small file problem
Register: opensourcedatasummit.com/
#OSDS2025 #ApacheHudi #DataLakehouse
Apache Hudi retweeted
๐กโจ [New Blog] Part 1 of a twoโpart deep dive on how Hudi brings multimodal indexing to the lakehouse.
๐ Read: hudi.apache.org/blog/2025/10โฆ
@apachehudi brings indexing to the data lakehouse with a selfโmanaged metadata table โ a multimodal index that tracks partition and file lists, column stats, and partition stats, using HFile for fast key lookups.
How data skipping works:
๐ Parse predicates from your SQL
โ๏ธ Prune partitions with partitionโlevel stats
โ๏ธ Prune files with fileโlevel stats
๐ Scan only the minimum required data
Why it matters: ๐ less I/O, โก faster queries, ๐ต lower compute โ at scale.
Pro tip โ๏ธ: index only the columns you frequently filter on using ๐ต๐ผ๐ผ๐ฑ๐ถ๐ฒ.๐บ๐ฒ๐๐ฎ๐ฑ๐ฎ๐๐ฎ.๐ถ๐ป๐ฑ๐ฒ๐
.๐ฐ๐ผ๐น๐๐บ๐ป.๐๐๐ฎ๐๐.๐ฐ๐ผ๐น๐๐บ๐ป.๐น๐ถ๐๐ to control metadata overhead.
#ApacheHudi #DataLakehouse #DataEngineering
Apache Hudi retweeted
[Blog] ๐๐๐๐ค๐ ๐๐จ๐ง๐ง๐๐๐ญ ๐ฏ๐ฌ. ๐
๐ฅ๐ข๐ง๐ค ๐ฏ๐ฌ. ๐๐ฉ๐๐ซ๐ค for data ingestion. Which should you choose?
โฆ ๐๐ฉ๐๐๐ก๐ ๐๐๐๐ค๐ ๐๐จ๐ง๐ง๐๐๐ญ: The reliable data mover ๐ As a component of Kafka, Connect is a framework for reliably streaming data between Kafka and other data systems. It uses a distributed and scalable architecture with source connectors to ingest data into Kafka and sink connectors to move data out to destinations like data lakehouse.
โฆ ๐๐ฉ๐๐๐ก๐ ๐
๐ฅ๐ข๐ง๐ค: The real-time specialist โก Flink is a true stream-processing engine that excels at low-latency, event-driven pipelines. A key feature is Flink CDC, which can ingest Change Data Capture (CDC) events directly from databases without needing Kafka. I recommend Flink when you need to filter, enrich, or run complex aggregations in-stream before loading.
โฆย ๐๐ฉ๐๐๐ก๐ ๐๐ฉ๐๐ซ๐ค: The batch powerhouse ๐ญ Spark is the go-to for large-scale batch processing and complex transformations. While it supports streaming, its micro-batch model means slightly higher latency. It's the ideal choice for heavy-duty batch backfills, ML data prep, and workloads where transformation complexity is more important than ultra-low latency.
In short, my recommendation is:
Just need to move data reliably? โก๏ธ Kafka Connect
Need true stream processing with low latency? โก๏ธ Flink
Need to process large batches or run complex transformations? โก๏ธ Spark
@apachehudi supports all three of these engines for writes. The right tool really depends on your specific needs for latency, processing, and scalability.
Read the full blog here: onehouse.ai/blog/kafka-conneโฆ
#ApacheKafka #ApacheFlink #ApacheSpark #ApacheHudi #DataLakehouse
Apache Hudi retweeted
[Blog] Struggling with Apache Iceberg performance when your data dimensions get too hot? ๐ฅ๐ก๏ธ
Frequent updates and deletes in Iceberg can lead to a "chilly meltdown," forcing a tough choice between fast writes and efficient reads. ๐ฅถ But what if you didn't have to compromise? ๐ค
In this recent blog, I explored how you can get the best of both worlds by combining the power of @apachehudi with @apachextable to serve fast, native Iceberg tables.
Read the full post to learn how to get fast writes and reads with Iceberg:
๐ onehouse.ai/blog/how-to-saveโฆ
#ApacheIceberg #ApacheHudi #ApacheXTable #DataLakehouse
Apache Hudi retweeted
Had a fantastic time speaking at the Data Streaming Summit SF 2025 this Tuesday! ๐ค It was a great pleasure to share my perspective on the architectural challenges of streaming writes in the data lakehouse.
I really enjoyed diving into how @apachehudi streaming-first designs for building streaming lakehouses. For those who couldn't make it, hereโs a quick recap of the key challenges and how Hudi handles them: ๐ก
โก๏ธ High-frequency updates: use ๐๐๐ซ๐ ๐-๐จ๐ง-๐๐๐๐ (๐๐๐) tables to absorb changes.
๐ Large volumes of mutable workloads: use ๐ซ๐๐๐จ๐ซ๐-๐ฅ๐๐ฏ๐๐ฅ ๐ข๐ง๐๐๐ฑ๐ข๐ง๐ to find records fast and keep write latency low.
๐๏ธ Small File issues: use ๐๐ฎ๐ญ๐จ-๐๐ข๐ฅ๐ ๐ฌ๐ข๐ณ๐ข๐ง๐ and ๐๐ฌ๐ฒ๐ง๐ ๐๐จ๐ฆ๐ฉ๐๐๐ญ๐ข๐จ๐ง/๐๐ฅ๐ฎ๐ฌ๐ญ๐๐ซ๐ข๐ง๐ .
โ๏ธ Conflicts and retries with OCC: Hudi 1.0 introduced ๐๐จ๐ง-๐๐ฅ๐จ๐๐ค๐ข๐ง๐ ๐๐จ๐ง๐๐ฎ๐ซ๐ซ๐๐ง๐๐ฒ ๐๐จ๐ง๐ญ๐ซ๐จ๐ฅ (๐๐๐๐) to avoid costly retries.
๐ Long commit history: Hudi 1.0 ๐๐๐ ๐๐ข๐ฆ๐๐ฅ๐ข๐ง๐ for optimized access and storage.
Thanks again to the Data Streaming Summit by @streamnativeio for having me!
#DataLakeHouse #DataStreaming #ApacheHudi #DSSSF25
Apache Hudi retweeted
๐๏ธ OLD NEWS: but worth a shout-out. Keys are optional in Hudi ...
One of Hudiโs core goals was to remove friction from building data lakes. Thatโs why โ for a long time now โ Hudi has quietly supported auto-generation of record keys. No need to think up a key field just to get started.
Old news, but worth reiterating. Shiyan Xu has a great blog walking through the considerations:
๐ hudi.apache.org/blog/2025/09โฆ
Why does this matter?
โขKeys are what enable Hudiโs unique features: indexing, seamless CDC, async & bounded compaction, log compactionโฆ all without extra work from you.
โขThey come at almost zero storage cost โ even more compressible than popular UUID formats.
โขThey lower the barrier to adoption for append-only workloads while keeping all of Hudiโs advanced table services intact.
Enjoy your table...
Apache Hudi retweeted
New work on Generic Table APIs from Snowflake and Onehouse in Polaris uses XTable for non-iceberg tables. Running XTable via REST API =
POST
/v1/conversion/table/
{"source-format": "HUDI | DELTA"
...}
Decoupling catalog spec from formats future proofs innovations in storage
Apache Hudi retweeted
I'm delighted to be delivering this talk at Data Streaming Summit SF 2025 hosted by @streamnativeio. Some highlights of the talk are:
โ
Understand the architecture and workflow of NBCC and its alignment with Flink for streaming ingestion.
โ
Learn how the new file layout, LSM-tree backed event logs, and TrueTime semantics eliminate ingestion bottlenecks.
โ
Gain actionable insights into using NBCC for concurrent, conflict-free ingestion with tunable consistency.
Check out more about the event: datastreaming-summit.org/eveโฆ
#ApacheFlink #ApacheHudi #DataStreamingSummit
Apache Hudi retweeted
โ๏ธ [Blog] Part 2 of building a RAG-based AI recommender ๐ค
A reliable, efficient data lakehouse isn't just a prerequisiteโit's the core component that enables RAG (Retrieval-Augmented Generation) applications to thrive. While the LLM usually gets the spotlight, the data pipeline does the heavy lifting.
Part 1 of the blog introduced RAG from a conceptual perspective and explore why solid data architecture is crucial for AI success.
In part 2, we go through each of the five main components in the demo app, and discuss the key code snippets. A complete working project is also shared.
๐ blog.datumagic.ai/p/buildingโฆ
The demo app uses @FastAPI for building the main server app, @apachehudi as the primary data source for incremental processing, @qdrant_engine for storing vectorized data for building context, and an @OpenAI model for generating responses.
#datalakehouse #ai #rag
Apache Hudi retweeted
๐ A new chapter "Running Hudi in Production" is now available in the early release of "Apache Hudiโข: The Definitive Guide" - the first book ever written about @apachehudi, by industry experts: @_xushiyan, Prashant Wason, @SudhaSakthee, and @RebeccaBilbro.
๐ Get a free copy of the e-book: onehouse.ai/whitepaper/apachโฆ
In this new chapter, you'll learn about
โ
Operational utilities provided by Hudi
โ
Hudi's platform integration features
โ
Tips on performance tuning
Apache Hudi transforms your data lake into a powerful lakehouse with database-like capabilities - efficient upserts, deletions, and incremental processing without the complexity you'd expect.
What's your biggest data lake challenge today? Drop a comment below - we'd love to hear your experiences!
#ApacheHudi #DataLake #DataEngineering
Apache Hudi retweeted
๐ก ๐๐ผ๐ ๐ฑ๐ผ๐ฒ๐ ๐ฑ๐ฎ๐๐ฎ-๐๐ธ๐ถ๐ฝ๐ฝ๐ถ๐ป๐ด ๐๐ผ๐ฟ๐ธ ๐ถ๐ป @apachehudi ?
The metadata table (a multi-modal index system) located within your Hudi table contains multiple index components such as ๐๐ถ๐น๐ฒ๐, ๐ฃ๐ฎ๐ฟ๐๐ถ๐๐ถ๐ผ๐ป ๐ฆ๐๐ฎ๐๐, ๐๐ผ๐น๐๐บ๐ป ๐ฆ๐๐ฎ๐๐.
โฆ ๐๐ถ๐น๐ฒ๐: tells you what partitions are there, and what files are under each partitions
โฆ ๐ฃ๐ฎ๐ฟ๐๐ถ๐๐ถ๐ผ๐ป ๐ฆ๐๐ฎ๐๐: tells you, for each partition, what are the min/max values for column A, B, C, ...
โฆ ๐๐ผ๐น๐๐บ๐ป ๐ฆ๐๐ฎ๐๐: tells you, for each data file, what are the min/max values for column A, B, C, ...
These indexes work jointly to reduce the files needed to be scanned by query engines:
1๏ธโฃ Query engine parses a SQL and extract the predicates from the WHERE clause
2๏ธโฃ Ask ๐๐ถ๐น๐ฒ๐ for a list of partitions, then prune it based on ๐ฃ๐ฎ๐ฟ๐๐ถ๐๐ถ๐ผ๐ป ๐ฆ๐๐ฎ๐๐
3๏ธโฃ For the pruned partition list, ask ๐๐ถ๐น๐ฒ๐ for a list of data files, then prune it based on ๐๐ผ๐น๐๐บ๐ป ๐ฆ๐๐ฎ๐๐
4๏ธโฃ Return the pruned data file list back to query engines for further planning
A powerful indexing system is the key to your performant queries!
#apachehudi #SQL #datalake
Apache Hudi retweeted
๐ New Blog: Building a RAG-based AI Recommender (Part 1/2)
๐ blog.datumagic.ai/p/buildingโฆ
๐ What's inside:
โฆ How RAG works end-to-end (chunking โ embedding โ retrieval โ generation)
โฆ Why 70% of AI success is actually data engineering ๐
โฆ How @apachehudi 's incremental processing keeps your AI current
โฆ The "garbage in, garbage out" reality that still haunts AI projects
๐ก Key insight: There's no shortcut to AI success. A solid data platform serving high-quality data is the prerequisite to unlocking AI's power.
๐ง Coming in Part 2: Hands-on implementation with @apachehudi , @qdrant_engine , and @FastAPI - building a complete AI recommender from scratch.
#AI #DataEngineering #ApacheHudi
Apache Hudi retweeted
I had the privilege of presenting Hudi-rs at VeloxCon 2025 - the native Rust implementation of @apachehudi that's opening new doors for cross-language data processing.
๐จ By rebuilding Hudi's core in Rust, we've created a foundation that supports multiple language bindings while maintaining consistent semantics.
๐ฆ We have integrated with Rust and Python frameworks like @ApacheDataFusio , @daft_dataframe , @raydistributed . Now we are adding APIs to work with C++ based frameworks like @velox_lib .
These integrations mean that regardless of your framework of choice, you can leverage Hudi's powerful transactional capabilities and table formats with native performance.
๐บ Check out the full talk here: youtu.be/N5RIWNEIBNQ
Interested in contributing? Check out the project at github.com/apache/hudi-rs
#apachehudi #rust #python #cpp #database #dataengineering #dataecosystem



























