@byte_array
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
Founder @Onehousehq — Data Infra for Agents. Creator of @apachehudi, built the first #DataLakehouse @Uber. Databases, Dist. Systems. LinkedIn, Confluent.
Joined April 2009
- Tweets1.4K
- Following234
- Followers1.9K
- Likes817
Approaching now 10 yrs in production at some of the largest data lakes in the world.
Collective accomplishments of the Apache Hudi community. 🫶
Uber published a teardown on cutting storage and compute costs for export workloads on GCS-backed Hudi tables.
The problem: DSAR requests and compliance queries pull a few records from years of history. Small result sets, but they scan the entire table repeatedly.
Repeated access breaks GCS auto-tiering. Data that should drop into Nearline, Coldline, or Archive stays hot in Standard. Storage costs stay high. Egress stays high. Metadata operations pile up.
Secondary indexes don't fix this. Target records scatter across most files since the query is still a scan. The index still points to a large file set. The engine still pays the scan cost.
⚡ What works: Hudi column stats + table sorting on the predicate column.
1️⃣ Column stats live in Hudi's metadata table, not Parquet footers. Stats are indexed for fast access for engine pruning. Cold files stay cold without incurring penalties. Auto-tiering works.
2️⃣ Sorting clusters matching records into a small file subset
Queries filter on rider_id or driver_id consistently. Sorting on that column tightens min/max ranges. Pruning gets dramatically more selective.
Uber sorted one partition and benchmarked:
📉 Files touched: 100% → 8%
📉 GCS egress: ↓92%
📉 Compute cost: ↓91%
📉 Storage footprint: ↓15% (clustering similar values improves compression)
Export workloads break typical lakehouse assumptions: narrow output, broad scan surface. Repeated execution keeps data hot, drives up costs across storage, retrieval, metadata ops, compute.
Column stats + sorting fixes it. Files stay cold. Queries stay cheap.
Lays out how it all connects
Your lakehouse has a new job: fueling open models.
Today we're launching AI training data pipelines on Onehouse for @baseten, @FireworksAI_HQ, and @togethercompute.
The problem: Most teams "version" training data by dumping JSONL files into cloud storage. When labels change or eval surfaces a gap, there's no way to trace examples back to source records or rebuild the dataset.
What we built:
✅ Quanton curates training examples from lakehouse tables using Spark SQL—join conversations to outcomes, filter by quality, validate schema, export versioned datasets.
✅ Onehouse submits training jobs to Baseten, Fireworks, or Together AI and tracks status. The provider returns fine-tuned weights or adapters.
✅ Lakegres serves point-in-time context during training or eval—queries tables for exact documents and state needed to replay traces.
How it works: OneFlow ingests source data → Quanton curates examples → Airflow coordinates → provider trains → Lakegres serves context.
The dataset stays reproducible. Engineers can inspect selection queries, trace examples to sources, and rebuild when labels change.
Go to quanton.dev and give it a spin.
🔥 📣 Your lakehouse already has the data. Your training pipeline doesn't know how to use it. We're announcing AI training data pipelines on Onehouse for @baseten , @FireworksAI_HQ , and @togethercompute
Training data curation today feels like carrying lake water in a leaky bucket. Most teams "version" training inputs by dumping JSONL files into a cloud folder. When a label changes or an evaluation surfaces a gap, there's no way to trace which examples came from where, or rebuild the dataset from updated source tables.
Consider a support agent learning from past conversations. Useful examples require joining each conversation to its outcome: Was the issue resolved? Was the answer correct? Did someone intervene?
The conversation alone isn't enough. The evidence lives in operational tables. When the model handles a class of requests poorly, the next training dataset needs to reflect updated labels, corrected data, and new selection logic. Teams need the selection query, the source-table versions it read, and the history of those corrections.
🔄 These are familiar data engineering requirements: joins, versioning, lineage, backfills. The lakehouse already handles them.
Here's what we built:
1️⃣ Quanton curates examples from lakehouse tables using SQL selection logic—join conversations to resolution status, filter by outcome quality, batch vector search for good/bad examples at scale. It writes versioned exports as training datasets.
2️⃣ Onehouse submits the training job to Baseten, Fireworks, or Together AI and tracks status in the UI. The provider trains the model and returns the fine-tuned weights or adapter.
3️⃣ Lakegres serves point-in-time context during training or eval—queries lakehouse tables for the exact documents, records, or state needed to replay a past trace.
The dataset remains reproducible. Engineers can inspect the selection query, trace examples to source records, and rebuild the dataset when labels change or evaluation surfaces gaps.
Go to quanton.dev for a free account and give it a spin, let us know!
Full workflow: OneFlow ingests source data → Quanton curates training examples → Airflow coordinates the pipeline → provider trains → Lakegres serves context.
onehouse.ai/blog/announcing-…
Companies spend billions on Spark compute every year. Most assume open-source accelerators can close the gap to commercial engines.
We ran TPC-DS 10 TB across 5 engines on the same 11-node cluster to find out.
OSS Spark: 12,200s (baseline)
Comet (tuned): 9,122s (25% faster)
Gluten (tuned): 8,563s (30% faster)
Databricks Photon 18.2: 2,550s (5× faster)
Quanton: 2,384s (5× faster)
Open-source accelerators can get you partway there—under ideal conditions.
The gap between OSS and commercial Spark isn’t just raw speed. It’s how things behave when you run a setup close to production scale.
⚠️ On a realistic memory:disk ratio, both Comet and Gluten fail with OOMs on q67 and q93. Commercial engines handle these workloads without tuning.
⚠️ 1 TB is often too small to understand performance profiles of production Spark/lakehouse setups, since most joins at that scale become broadcast joins on the TPC-DS schema.
#ApacheSpark #DataEngineering
quanton.dev/benchmarks/ has more writer/ETL benchmarks for those thinking beyond tpc-ds.
The 30x jump in open-model adoption this year hides a costly blind spot.
As Meta, GLM, Kimi, Qwen, and Mistral models get more capable, more teams are customizing them. But SFT, preference tuning, and RFT aren't interchangeable—and the differences determine whether you spend $50 or $50,000 for the same result.
1️⃣ Supervised fine-tuning (SFT)
You provide prompt/response pairs. The model learns by imitation.
Think: prompt → ideal response
2️⃣ Preference tuning (DPO)
For each prompt, provide multiple responses and mark which is better. The model learns from comparisons.
Think: prompt → preferred response + rejected response
3️⃣ Reinforcement fine-tuning (RFT)
The model produces an answer, a grader scores it, and that becomes the reward signal. The grader can verify correctness, run tests, or apply domain rules.
Think: task → response → grader → reward
📊 Each approach demands different data.
SFT needs accurate, representative prompt/response pairs with consistent formatting.
Preference tuning needs the full comparison record: what outputs were judged, which won, and the context.
RFT needs a representative task distribution, execution context, and clear lineage to the source.
⚠️ The bottleneck across all three: data management.
Training data is a data engineering problem:
🔍 Curate datasets from production
✅ Validate before training
🔗 Track provenance for every example
📦 Version each dataset
🔄 Make every run reproducible
The training API is the easy part. The leverage is in the data infrastructure underneath.
Turns out AWS pricing is negotiable if you blog about it. AWS S3 Tables launched at $5.04 to compact 100GB Iceberg . EMR: $0.17. 29× markup.
When we published the teardown, AWS dropped the price by 10× right after. Then it was $0.52 for the same job. EMR still 3× cheaper. Who knew transparency was a feature request?
AWS has so many amazing services. But, 2 years in, S3 Tables still ships with the same foundational limitations it had at launch. Sadly, still only seems to beat competition by calling customer execs with some FUD about risk and support gaps if they don't adopt the managed service.
Price cuts are good. Price cuts after being caught aren't really a win. They just make you wonder what else sold this way.
Blog 2/price drop: AWS's S3 Tables pricing update: aws.amazon.com/about-aws/wha…
Blog 3/still expensive & slow: Full cost teardown: onehouse.ai/blog/aws-s3-tabl…
Vinoth Chandar retweeted
"Async compaction" is easy to claim. Building one that doesn't stall ingestion is much harder.
Every mature database figured this out decades ago.
✅ Foreground writes establish correctness.
⚙️ Background maintenance restores efficiency.
The two shouldn't block each other.
As lakehouses evolve from batch-oriented systems to continuously mutating platforms powering CDC, streaming, and AI, that architectural separation matters more every year.
My latest blog explores why background maintenance isn't just another feature—it's an architectural primitive.
👇🔗 hudi.apache.org/blog/2026/07…
#ApacheHudi #Lakehouse #DataEngineering #StorageSystems #DistributedSystems #OpenSource
Most AI engineers describe fine-tuning as: make a file, hit a training endpoint.
What I saw at Uber and LinkedIn: ETA prediction and People You May Know didn't win on just better models. They won by nailing the full data lifecycle:
Operational data → reliable training data → better models → better products → better outcomes → more operational data.
✅ The advantage: infrastructure that democratized access to signals and experimentation. Better models powered better products, whose usage generated even better data.
🔄 AI training data works the same way. Examples arrive continuously. Labels get corrected. Eval failures create hard negatives. Customer feedback shifts preferences. Policies require deletion. Parsers, pipelines, and models change the derived data.
The model trains periodically. The data changes constantly—and it's mutable.
🛠️ That requires production data discipline:
• Canonical records and derived views
• Incremental updates and backfills
• Reproducible snapshots
• Lineage and quality checks
• Delete propagation and retention
• Schema evolution and observability
⚠️ Consider one mislabeled support conversation. Fixing it means updating the source, regenerating training examples, revising eval data, refreshing embeddings, updating hard-negative pools—across every snapshot.
Deleting won't unlearn what the model saw. But if we can't reliably remove it from future training, evaluation, and retrieval, we don't control the lifecycle.
🔁 AI teams are rediscovering database problems: mutable state, consistency, lineage, backfills, deletes, retention, reproducibility, derived-data management.
The final training file may look simple. The system that makes it trustworthy is not.
Apache XTable 0.4.0 (incubating) is out. 45 commits from 19 contributors, including patches from companies running it in production.
Format support went from 3 to 5. Paimon and Parquet join Hudi, Iceberg, and Delta Lake. Still zero data rewritten—just metadata conversion.
What shipped:
🔧 Spark runtime jar that reuses your cluster's existing Hudi/Iceberg/Delta libraries. Add it with --jars, call XTableSyncService after your write, or submit XTableSparkSync standalone. Supports Spark 3.4 and 3.5.
🔗 REST service for format translation, integrates with Polaris for catalog-aware sync.
⚡ Delta Kernel source and target—read and write Delta tables through delta-kernel-api without needing a SparkSession.
Project's been quiet, but that changes. We plan to contribute upstream features around data governance and semantic layers in the coming months.
Download: xtable.apache.org/downloads
Release notes: xtable.apache.org/blog/apach…
Your lakehouse already has the data. Transactions, events, customer records, inventory updates. You spent millions ingesting it.
But that same data could power AI agents reasoning over live business state—if the vectors stayed fresh.
⚡ NielsenIQ's workload hits single-digit second latencies for text search on Hudi/Delta lakehouses.
Leveraging record-level indexing to fetch matching records, is now lowering this to few milliseconds
#VectorSearch #Lakehouse
Are you generating vector embeddings at scale on your data? how? what are the challenges?
[1] Nielsen post linkedin.com/pulse/text-base…
[2] Vector search on the lake hudi.apache.org/blog/2026/07…