Everett Roeth (@everettroeth) profile picture

Building stuff @LakeSail

California, USA
Joined July 2020
Everett Roeth retweeted
Why is Apache Arrow becoming the universal columnar foundation for modern data and AI infrastructure? Apache Arrow is great because of its language-independent columnar memory format, zero-copy data interchange, efficient in-memory processing, and broad multi-language ecosystem. That’s why projects like Apache Spark, Apache Parquet, ClickHouse, Dremio, Polars, Pandas, InfluxData, OpenObserve, Greptime, LakeSail, Spice AI, IOMETE, Daft, Apache Doris, and so many other great systems use it. For data and AI systems, Arrow is not just another format. It is becoming a foundational layer for moving and processing data efficiently across the entire ecosystem. RisingWave, a unified data platform for real-time and Iceberg-based lakehouse workloads, also uses Arrow extensively, particularly for interoperability with Apache Iceberg, Apache DataFusion, Parquet, UDFs, and some connectors. Shoutout to our partner @columnar_tech for advancing Apache Arrow and ADBC. They're really doing great work and redefining database access and connectivity. 🙌
1
3
1
6
590
Wow. Future of computing!
LakeSail announced a strategic partnership with Oxide Computer Company to help enterprises modernize data and AI infrastructure while maintaining control over security, performance, and cost. The partnership follows the successful validation of Sail on Oxide’s platform. Sail, LakeSail’s open-source, Rust-native, Spark-compatible compute engine, ran on Oxide’s integrated, rack-scale cloud platform, establishing a validated foundation for secure, customer-controlled AI environments. Oxide delivers a cloud-like experience within a customer’s data center through an integrated platform and unified control plane. Its architecture includes a hardware root of trust and mechanisms for verifying the integrity of the software stack. Sail preserves Spark APIs, SQL, and DataFrames while replacing the traditional JVM execution layer with a Rust-native architecture. It supports Python, Apache Iceberg, and Delta Lake, allowing enterprises to modernize without rewriting existing applications. By eliminating the JVM execution layer, Sail reduces costs and performance bottlenecks associated with cluster startup, garbage collection, memory tuning, idle capacity, and overprovisioning. Depending on workload and configuration, LakeSail has demonstrated performance improvements of up to 10x and infrastructure-cost reductions of up to 98% compared with equivalent Spark and JVM-based deployments. Enterprises can use Sail as open-source software or purchase the commercial LakeSail Platform for management, governance, security, and support capabilities. “We believe data processing and AI belong in the same engine rather than in separate systems customers must stitch together,” said Shehab Amin, Co-Founder and CEO of LakeSail. “Combining Sail’s efficiency with Oxide’s customer-owned cloud infrastructure gives enterprises more freedom to build without inheriting more complexity or traditional JVM economics.” “As data management and processing become more central to AI, we're excited to see LakeSail running on Oxide," said Steve Tuck, Co-Founder and CEO of Oxide Computer Company. "Running LakeSail's modern execution engine on Oxide's on-prem Cloud Computer allows customers to gain insights from their data faster while maintaining the control over where that data lives." Read the full press release: lakesail.com/blog/oxide-part…
1
3
146
The other day my brokerage account rep recommended me to activate "voice authentication" claiming it's a "very secure new technology", obviously just a new feature she's prompted to tell customers... Needless to say, I couldn't be more skeptical having my "voice" trained on like a few minutes of conversation grant access to my financial accounts. block.xyz/dessa
13
cmux is by far the best agentic coding terminal I've come across to date. highly recommend.
1
40
Everett Roeth retweeted
A recent Databricks customer's workload ran 381x faster: 38 minutes to 6 seconds. Test yours in minutes.

3
45
1,441,723
Everett Roeth retweeted
A data platform that's simple, transparently priced, and ridiculously fast. The LakeSail Platform runs your Spark SQL and Python jobs on a Rust-native runtime. It's faster than Spark, drastically cuts hardware costs, and scales from a single node to a multi-node cluster. It reads and writes your Delta Lake and Iceberg tables where they already live, so your lakehouse stays open and your data stays in place. For those who want to run Sail in production most efficiently, with all the operations handled for you, this platform was built for you.
1
2
1
6
313
Everett Roeth retweeted
We just published a derived TPC-DS for Sail at 3 TB and 1 TB. Here is a comparison of our benchmark to another independently published benchmark on Spark and Spark + Gluten/Velox by AWS.
1
2
6
586
The data space has changed. If you're running Spark on Databricks, AWS Glue, or any other managed Spark service, this is well worth your time to check out.
A data platform that's simple, transparently priced, and ridiculously fast. The LakeSail Platform runs your Spark SQL and Python jobs on a Rust-native runtime. It's faster than Spark, drastically cuts hardware costs, and scales from a single node to a multi-node cluster. It reads and writes your Delta Lake and Iceberg tables where they already live, so your lakehouse stays open and your data stays in place. For those who want to run Sail in production most efficiently, with all the operations handled for you, this platform was built for you.
1
1
2
45
Everett Roeth retweeted
Sail ran derived TPC-DS at 3TB and 1TB across a distributed cluster, and the results are the latest evidence for our vision in rethinking distributed compute. Hadoop pioneered open-source distributed processing, built around disk. Spark moved intermediate data into memory, though its shuffle stayed disk-first. When we started Sail, we envisioned another shift in distributed compute, toward doing even more of the work in memory, with an engine built entirely in Rust. We built the first version of Sail's distributed architecture in November 2024, with Arrow Flight carrying shuffle data between workers, and redesigned the cluster control plane earlier this year. Arrow Flight shuffle can look surprising at first. A TPC-DS query with dozens of stages and a few hundred partitions can have more than a million logical shuffle streams active in the cluster. Arrow Flight is built on gRPC and HTTP/2, so one TCP connection per pair of workers carries them all. The same approach has since come up in the DataFusion Comet and Riffle communities as well, and we are glad to have started proving this direction two years ago. The full benchmark ran in 1,289 seconds at 3TB and 444 seconds at 1TB, with no shuffle data written to disk. On similar hardware and settings, Sail outperforms the published TPC-DS results for Spark and Spark accelerators at these scales. The post covers the results, the design, and the full setup: lakesail.com/blog/tpcds-benc…
1
2
4
137
Everett Roeth retweeted
Sail is the first open-source engine to ship with built-in Jev support! Check out the docs to get started: docs.lakesail.com/sail/lates…
3
8
11,155
This is a fascinating presentation.
Adyen’s data team explored DuckDB and Polars but landed on Sail as a new engine. They explain their reasoning in a recent presentation they gave at PyData Amsterdam.
3
37
Two results their Principal Engineer, Santosh, shared: • A CI test suite went from 1,335 seconds on Spark to 78.44 seconds on Sail, roughly 17x faster. • An Airflow job's median runtime dropped from about 57 minutes to 8.5 minutes, using fewer cores and less memory.
1
3
Everett Roeth retweeted
Sail now includes Jev built natively into the engine, the first of its kind. Any Spark SQL or DataFrame query can ask Jev about every row of a table, with nothing extra to install. The answers come back as typed columns you can filter, join, and aggregate like any other data. The same query can run on your laptop or across a cluster. Check out the full release and some examples of it in action. Link in thread.
1
3
7
259
Everett Roeth retweeted
Sail 0.7.1 adds the first ever Rust implementation of Apache Celeborn as a shuffle backend. If you already run Celeborn for your Spark workloads, Sail uses that cluster as it stands. The same masters, same workers, nothing is changed on the Celeborn side. Read more on the engineering in our blog: lakesail.com/blog/sail-celeb…
2
4
154
All ahoy— @LakeSail team is coming to Amsterdam! Speaking at @pydataamsterdam and our own meetup with @adyen on Wednesday: bay.news/amsterdam1 — join us for food, drink, Rust, AI and Data!
1
6
271
Everett Roeth retweeted
The interesting part of rebuilding Spark in Rust wasn’t the Rust. @shehab_amins joined Daniel Beach on Data Engineering Central to talk about why Spark compatibility turned out to be much harder, and more interesting, than expected. And what changes when the thing querying your lakehouse is an agent instead of an analyst. Delta vs. Iceberg, streaming vs. batch, Arrow and DataFusion. Full episode below. dataengineeringcentral.subst…
3
4
6
707
Everett Roeth retweeted
Your query plan, before and after checkpoint.
1
2
4
356