Apache Hudi (@apachehudi) header
Apache Hudi (@apachehudi) profile picture

Official twitter handle of Apache Hudi, an open data lakehouse platform. https://nitter.cf/t.co/SXay7oHNah

Joined January 2019
Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)Photo by Apache Hudi (@apachehudi)
Apache Hudi retweeted
At scale, table metadata explodes, as Google's VLDB paper "Big Metadata" shows. Hudi's metadata table treats it like big data ๐Ÿ“Š: self-managed, indexed Key bits: - Partition/file listings ๐Ÿ“ - Column stats ๐Ÿ“ˆ - Record indexes ๐Ÿ” This ensures efficient reads/writes as things grow ๐Ÿš€.
1
1
3
258
Apache Hudi retweeted
2/ Apache Hudi continues to operate at trillion-record scale inside one of the most demanding data platforms in the industry. Not experiments. Not edge cases. Production systems moving hundreds of petabytes. At Uber, Hudi is powering: โ€ข high-volume ingestion โ€ข continuous updates & merges โ€ข replication โ€ข incremental processing
1
2
5
403
Apache Hudi retweeted
[BLOG] Just published a blog on a problem that trips up many teams: retry storms from concurrent writers on data lakehouses Streaming ingestion + long-running maintenance jobs = OCC fails you. Long jobs lose every commit race and retry indefinitely @apachehudi NBCC solves this by eliminating conflicts by designโ€”writers append in parallel, completion time determines order. No retries needed. Deep dive ๐Ÿ‘‡ hudi.apache.org/blog/2025/12โ€ฆ #ApacheHudi #DataLakehouse #DataEngineering
1
2
277
Apache Hudi retweeted
The pace of innovation in the @apachehudi ecosystem is accelerating fast. If you want to keep up with these architectural shifts and production lessons, the monthly newsletter is the best way to stay plugged in. ๐Ÿ—ž๏ธ Read + subscribe: hudinewsletter.substack.com/โ€ฆ
1
1
1
192
Apache Hudi retweeted
New release: @apachehudi 1.1 is out! The standout innovation: Pluggable Table Format Framework decouples Hudiโ€™s storage engine from the format, unlocking full read compatibility with Iceberg and Delta Lake. Itโ€™s a massive unlock: Hudiโ€™s ingestion performance with total format interoperability. Read my full write-up here: hudi.apache.org/blog/2025/11โ€ฆ #ApacheHudi #DataLakehouse
1
1
6
220
Apache Hudi retweeted
๐ŸŽ‰ Apache Hudi 1.1 has landedโ€”and it's a banger! ๐Ÿš€ This release is packed with innovations that will transform how you build data lakehouses: ๐ŸŽฏ Pluggable Table Format Framework for multi-format support (Hudi, Iceberg, Delta Lake) โšก 15x faster clustering with Parquet binary copyโ€”yes, you read that right! โšก 2-3x higher throughput for Flink writers! ๐Ÿš€ Spark 4.0 & Flink 2.0 support โ€”harness the latest engine capabilities Read the blog: hudi.apache.org/blog/2025/11โ€ฆ #ApacheHudi #ApacheSpark #ApacheFlink
1
3
1
8
1,133
Apache Hudi retweeted
๐Ÿ“š The Definitive Guide of @apachehudi has arrived! This was a collaborative effort with my co-authors Prashant, @SudhaSakthee , and @RebeccaBilbro , bringing together years of collective knowledge from the Apache Hudi community โ€” from fundamental concepts to production patterns for building data lakehouses. Grateful to @byte_array for writing the foreword, and to all the tech reviewers, editors, and those who provided praise and support for the book. Thanks to @OReillyMedia and @Onehousehq for making this happen, and to everyone in the Hudi community who's shaped this technology. For anyone building data lakehouses or exploring Hudi, hope this guide proves useful. ๐Ÿš€ #ApacheHudi #DataEngineering #DataLakehouse
1
2
148
Apache Hudi retweeted
Speaking at Open Source Data Summit (Nov 13) on building streaming-first lakehouses with @apachehudi ๐ŸŽค We'll dive into: โ€ข Merge-on-Read tables โ€ข NBCC for conflict-free writes โ€ข LSM Timeline โ€ข Solving the small file problem Register: opensourcedatasummit.com/ #OSDS2025 #ApacheHudi #DataLakehouse
3
2
184
Apache Hudi retweeted
๐Ÿ’กโœจ [New Blog] Part 1 of a twoโ€‘part deep dive on how Hudi brings multimodal indexing to the lakehouse. ๐Ÿ”— Read: hudi.apache.org/blog/2025/10โ€ฆ @apachehudi brings indexing to the data lakehouse with a selfโ€‘managed metadata table โ€” a multimodal index that tracks partition and file lists, column stats, and partition stats, using HFile for fast key lookups. How data skipping works: ๐Ÿ”Ž Parse predicates from your SQL โœ‚๏ธ Prune partitions with partitionโ€‘level stats โœ‚๏ธ Prune files with fileโ€‘level stats ๐Ÿ“‰ Scan only the minimum required data Why it matters: ๐Ÿ“‰ less I/O, โšก faster queries, ๐Ÿ’ต lower compute โ€” at scale. Pro tip โš™๏ธ: index only the columns you frequently filter on using ๐—ต๐—ผ๐—ผ๐—ฑ๐—ถ๐—ฒ.๐—บ๐—ฒ๐˜๐—ฎ๐—ฑ๐—ฎ๐˜๐—ฎ.๐—ถ๐—ป๐—ฑ๐—ฒ๐˜….๐—ฐ๐—ผ๐—น๐˜‚๐—บ๐—ป.๐˜€๐˜๐—ฎ๐˜๐˜€.๐—ฐ๐—ผ๐—น๐˜‚๐—บ๐—ป.๐—น๐—ถ๐˜€๐˜ to control metadata overhead. #ApacheHudi #DataLakehouse #DataEngineering
1
1
1
175
Apache Hudi retweeted
[Blog] ๐Š๐š๐Ÿ๐ค๐š ๐‚๐จ๐ง๐ง๐ž๐œ๐ญ ๐ฏ๐ฌ. ๐…๐ฅ๐ข๐ง๐ค ๐ฏ๐ฌ. ๐’๐ฉ๐š๐ซ๐ค for data ingestion. Which should you choose? โœฆ ๐€๐ฉ๐š๐œ๐ก๐ž ๐Š๐š๐Ÿ๐ค๐š ๐‚๐จ๐ง๐ง๐ž๐œ๐ญ: The reliable data mover ๐Ÿšš As a component of Kafka, Connect is a framework for reliably streaming data between Kafka and other data systems. It uses a distributed and scalable architecture with source connectors to ingest data into Kafka and sink connectors to move data out to destinations like data lakehouse. โœฆ ๐€๐ฉ๐š๐œ๐ก๐ž ๐…๐ฅ๐ข๐ง๐ค: The real-time specialist โšก Flink is a true stream-processing engine that excels at low-latency, event-driven pipelines. A key feature is Flink CDC, which can ingest Change Data Capture (CDC) events directly from databases without needing Kafka. I recommend Flink when you need to filter, enrich, or run complex aggregations in-stream before loading. โœฆย ๐€๐ฉ๐š๐œ๐ก๐ž ๐’๐ฉ๐š๐ซ๐ค: The batch powerhouse ๐Ÿญ Spark is the go-to for large-scale batch processing and complex transformations. While it supports streaming, its micro-batch model means slightly higher latency. It's the ideal choice for heavy-duty batch backfills, ML data prep, and workloads where transformation complexity is more important than ultra-low latency. In short, my recommendation is: Just need to move data reliably? โžก๏ธ Kafka Connect Need true stream processing with low latency? โžก๏ธ Flink Need to process large batches or run complex transformations? โžก๏ธ Spark @apachehudi supports all three of these engines for writes. The right tool really depends on your specific needs for latency, processing, and scalability. Read the full blog here: onehouse.ai/blog/kafka-conneโ€ฆ #ApacheKafka #ApacheFlink #ApacheSpark #ApacheHudi #DataLakehouse
1
1
126
Apache Hudi retweeted
[Blog] Struggling with Apache Iceberg performance when your data dimensions get too hot? ๐Ÿ”ฅ๐ŸŒก๏ธ Frequent updates and deletes in Iceberg can lead to a "chilly meltdown," forcing a tough choice between fast writes and efficient reads. ๐Ÿฅถ But what if you didn't have to compromise? ๐Ÿค” In this recent blog, I explored how you can get the best of both worlds by combining the power of @apachehudi with @apachextable to serve fast, native Iceberg tables. Read the full post to learn how to get fast writes and reads with Iceberg: ๐Ÿ‘‰ onehouse.ai/blog/how-to-saveโ€ฆ #ApacheIceberg #ApacheHudi #ApacheXTable #DataLakehouse
1
1
4
2,137
Apache Hudi retweeted
Had a fantastic time speaking at the Data Streaming Summit SF 2025 this Tuesday! ๐ŸŽค It was a great pleasure to share my perspective on the architectural challenges of streaming writes in the data lakehouse. I really enjoyed diving into how @apachehudi streaming-first designs for building streaming lakehouses. For those who couldn't make it, hereโ€™s a quick recap of the key challenges and how Hudi handles them: ๐Ÿ’ก โšก๏ธ High-frequency updates: use ๐Œ๐ž๐ซ๐ ๐ž-๐จ๐ง-๐‘๐ž๐š๐ (๐Œ๐Ž๐‘) tables to absorb changes. ๐Ÿ“ˆ Large volumes of mutable workloads: use ๐ซ๐ž๐œ๐จ๐ซ๐-๐ฅ๐ž๐ฏ๐ž๐ฅ ๐ข๐ง๐๐ž๐ฑ๐ข๐ง๐  to find records fast and keep write latency low. ๐Ÿ—‚๏ธ Small File issues: use ๐š๐ฎ๐ญ๐จ-๐Ÿ๐ข๐ฅ๐ž ๐ฌ๐ข๐ณ๐ข๐ง๐  and ๐š๐ฌ๐ฒ๐ง๐œ ๐œ๐จ๐ฆ๐ฉ๐š๐œ๐ญ๐ข๐จ๐ง/๐œ๐ฅ๐ฎ๐ฌ๐ญ๐ž๐ซ๐ข๐ง๐ . โš”๏ธ Conflicts and retries with OCC: Hudi 1.0 introduced ๐๐จ๐ง-๐๐ฅ๐จ๐œ๐ค๐ข๐ง๐  ๐‚๐จ๐ง๐œ๐ฎ๐ซ๐ซ๐ž๐ง๐œ๐ฒ ๐‚๐จ๐ง๐ญ๐ซ๐จ๐ฅ (๐๐๐‚๐‚) to avoid costly retries. ๐Ÿ“œ Long commit history: Hudi 1.0 ๐‹๐’๐Œ ๐“๐ข๐ฆ๐ž๐ฅ๐ข๐ง๐ž for optimized access and storage. Thanks again to the Data Streaming Summit by @streamnativeio for having me! #DataLakeHouse #DataStreaming #ApacheHudi #DSSSF25
1
3
189
๐Ÿ—ž๏ธ OLD NEWS: but worth a shout-out. Keys are optional in Hudi ... One of Hudiโ€™s core goals was to remove friction from building data lakes. Thatโ€™s why โ€” for a long time now โ€” Hudi has quietly supported auto-generation of record keys. No need to think up a key field just to get started. Old news, but worth reiterating. Shiyan Xu has a great blog walking through the considerations: ๐Ÿ‘‰ hudi.apache.org/blog/2025/09โ€ฆ Why does this matter? โ€ขKeys are what enable Hudiโ€™s unique features: indexing, seamless CDC, async & bounded compaction, log compactionโ€ฆ all without extra work from you. โ€ขThey come at almost zero storage cost โ€” even more compressible than popular UUID formats. โ€ขThey lower the barrier to adoption for append-only workloads while keeping all of Hudiโ€™s advanced table services intact. Enjoy your table...
1
1
263
Apache Hudi retweeted
New work on Generic Table APIs from Snowflake and Onehouse in Polaris uses XTable for non-iceberg tables. Running XTable via REST API = POST /v1/conversion/table/ {"source-format": "HUDI | DELTA" ...} Decoupling catalog spec from formats future proofs innovations in storage
1
3
4
333
Apache Hudi retweeted
I'm delighted to be delivering this talk at Data Streaming Summit SF 2025 hosted by @streamnativeio. Some highlights of the talk are: โœ… Understand the architecture and workflow of NBCC and its alignment with Flink for streaming ingestion. โœ… Learn how the new file layout, LSM-tree backed event logs, and TrueTime semantics eliminate ingestion bottlenecks. โœ… Gain actionable insights into using NBCC for concurrent, conflict-free ingestion with tunable consistency. Check out more about the event: datastreaming-summit.org/eveโ€ฆ #ApacheFlink #ApacheHudi #DataStreamingSummit
1
5
313
Apache Hudi retweeted
โœ๏ธ [Blog] Part 2 of building a RAG-based AI recommender ๐Ÿค– A reliable, efficient data lakehouse isn't just a prerequisiteโ€”it's the core component that enables RAG (Retrieval-Augmented Generation) applications to thrive. While the LLM usually gets the spotlight, the data pipeline does the heavy lifting. Part 1 of the blog introduced RAG from a conceptual perspective and explore why solid data architecture is crucial for AI success. In part 2, we go through each of the five main components in the demo app, and discuss the key code snippets. A complete working project is also shared. ๐Ÿ‘‰ blog.datumagic.ai/p/buildingโ€ฆ The demo app uses @FastAPI for building the main server app, @apachehudi as the primary data source for incremental processing, @qdrant_engine for storing vectorized data for building context, and an @OpenAI model for generating responses. #datalakehouse #ai #rag
2
4
427
Apache Hudi retweeted
๐ŸŽ‰ A new chapter "Running Hudi in Production" is now available in the early release of "Apache Hudiโ„ข: The Definitive Guide" - the first book ever written about @apachehudi, by industry experts: @_xushiyan, Prashant Wason, @SudhaSakthee, and @RebeccaBilbro. ๐Ÿ‘‰ Get a free copy of the e-book: onehouse.ai/whitepaper/apachโ€ฆ In this new chapter, you'll learn about โœ… Operational utilities provided by Hudi โœ… Hudi's platform integration features โœ… Tips on performance tuning Apache Hudi transforms your data lake into a powerful lakehouse with database-like capabilities - efficient upserts, deletions, and incremental processing without the complexity you'd expect. What's your biggest data lake challenge today? Drop a comment below - we'd love to hear your experiences! #ApacheHudi #DataLake #DataEngineering
2
3
362
Apache Hudi retweeted
๐Ÿ’ก ๐—›๐—ผ๐˜„ ๐—ฑ๐—ผ๐—ฒ๐˜€ ๐—ฑ๐—ฎ๐˜๐—ฎ-๐˜€๐—ธ๐—ถ๐—ฝ๐—ฝ๐—ถ๐—ป๐—ด ๐˜„๐—ผ๐—ฟ๐—ธ ๐—ถ๐—ป @apachehudi ? The metadata table (a multi-modal index system) located within your Hudi table contains multiple index components such as ๐—™๐—ถ๐—น๐—ฒ๐˜€, ๐—ฃ๐—ฎ๐—ฟ๐˜๐—ถ๐˜๐—ถ๐—ผ๐—ป ๐—ฆ๐˜๐—ฎ๐˜๐˜€, ๐—–๐—ผ๐—น๐˜‚๐—บ๐—ป ๐—ฆ๐˜๐—ฎ๐˜๐˜€. โœฆ ๐—™๐—ถ๐—น๐—ฒ๐˜€: tells you what partitions are there, and what files are under each partitions โœฆ ๐—ฃ๐—ฎ๐—ฟ๐˜๐—ถ๐˜๐—ถ๐—ผ๐—ป ๐—ฆ๐˜๐—ฎ๐˜๐˜€: tells you, for each partition, what are the min/max values for column A, B, C, ... โœฆ ๐—–๐—ผ๐—น๐˜‚๐—บ๐—ป ๐—ฆ๐˜๐—ฎ๐˜๐˜€: tells you, for each data file, what are the min/max values for column A, B, C, ... These indexes work jointly to reduce the files needed to be scanned by query engines: 1๏ธโƒฃ Query engine parses a SQL and extract the predicates from the WHERE clause 2๏ธโƒฃ Ask ๐—™๐—ถ๐—น๐—ฒ๐˜€ for a list of partitions, then prune it based on ๐—ฃ๐—ฎ๐—ฟ๐˜๐—ถ๐˜๐—ถ๐—ผ๐—ป ๐—ฆ๐˜๐—ฎ๐˜๐˜€ 3๏ธโƒฃ For the pruned partition list, ask ๐—™๐—ถ๐—น๐—ฒ๐˜€ for a list of data files, then prune it based on ๐—–๐—ผ๐—น๐˜‚๐—บ๐—ป ๐—ฆ๐˜๐—ฎ๐˜๐˜€ 4๏ธโƒฃ Return the pruned data file list back to query engines for further planning A powerful indexing system is the key to your performant queries! #apachehudi #SQL #datalake
1
3
4
309
Apache Hudi retweeted
๐Ÿš€ New Blog: Building a RAG-based AI Recommender (Part 1/2) ๐Ÿ‘‰ blog.datumagic.ai/p/buildingโ€ฆ ๐Ÿ“š What's inside: โœฆ How RAG works end-to-end (chunking โ†’ embedding โ†’ retrieval โ†’ generation) โœฆ Why 70% of AI success is actually data engineering ๐Ÿ“Š โœฆ How @apachehudi 's incremental processing keeps your AI current โœฆ The "garbage in, garbage out" reality that still haunts AI projects ๐Ÿ’ก Key insight: There's no shortcut to AI success. A solid data platform serving high-quality data is the prerequisite to unlocking AI's power. ๐Ÿ”ง Coming in Part 2: Hands-on implementation with @apachehudi , @qdrant_engine , and @FastAPI - building a complete AI recommender from scratch. #AI #DataEngineering #ApacheHudi
8
31
1,495
Apache Hudi retweeted
I had the privilege of presenting Hudi-rs at VeloxCon 2025 - the native Rust implementation of @apachehudi that's opening new doors for cross-language data processing. ๐Ÿ”จ By rebuilding Hudi's core in Rust, we've created a foundation that supports multiple language bindings while maintaining consistent semantics. ๐Ÿฆ€ We have integrated with Rust and Python frameworks like @ApacheDataFusio , @daft_dataframe , @raydistributed . Now we are adding APIs to work with C++ based frameworks like @velox_lib . These integrations mean that regardless of your framework of choice, you can leverage Hudi's powerful transactional capabilities and table formats with native performance. ๐Ÿ“บ Check out the full talk here: youtu.be/N5RIWNEIBNQ Interested in contributing? Check out the project at github.com/apache/hudi-rs #apachehudi #rust #python #cpp #database #dataengineering #dataecosystem
6
1
25
4,459