LakeSail

No JVM. 10x faster. 98% cost reduction.

Sail 0.7.3 is out. This release expands what you can do with Delta Lake and Apache Iceberg in Sail. We’ve added OPTIMIZE, VACUUM, and table history inspection. Iceberg gains SQL support for branches and tags, metadata tables, and procedures for compaction and snapshot management. There are also new SQL functions for vectors and input file metadata access. We also fixed many query planning issues found via differential testing against JVM Spark. Time zone handling, type coercion, scalar subqueries, set operations, aggregates, joins, and sorts now behave correctly in many complex edge cases. Thanks to everyone who contributed, including the eight first-time contributors in this release. Release notes: github.com/lakehq/sail/relea…

pip install pysail

LakeSail and Microsoft OneLake. Microsoft just shared a closer look at the integration, including how teams can work with OneLake data through familiar Spark SQL and DataFrame interfaces. community.fabric.microsoft.c…

LakeSail announced a strategic partnership with Oxide Computer Company to help enterprises modernize data and AI infrastructure while maintaining control over security, performance, and cost. The partnership follows the successful validation of Sail on Oxide’s platform. Sail, LakeSail’s open-source, Rust-native, Spark-compatible compute engine, ran on Oxide’s integrated, rack-scale cloud platform, establishing a validated foundation for secure, customer-controlled AI environments. Oxide delivers a cloud-like experience within a customer’s data center through an integrated platform and unified control plane. Its architecture includes a hardware root of trust and mechanisms for verifying the integrity of the software stack. Sail preserves Spark APIs, SQL, and DataFrames while replacing the traditional JVM execution layer with a Rust-native architecture. It supports Python, Apache Iceberg, and Delta Lake, allowing enterprises to modernize without rewriting existing applications. By eliminating the JVM execution layer, Sail reduces costs and performance bottlenecks associated with cluster startup, garbage collection, memory tuning, idle capacity, and overprovisioning. Depending on workload and configuration, LakeSail has demonstrated performance improvements of up to 10x and infrastructure-cost reductions of up to 98% compared with equivalent Spark and JVM-based deployments. Enterprises can use Sail as open-source software or purchase the commercial LakeSail Platform for management, governance, security, and support capabilities. “We believe data processing and AI belong in the same engine rather than in separate systems customers must stitch together,” said Shehab Amin, Co-Founder and CEO of LakeSail. “Combining Sail’s efficiency with Oxide’s customer-owned cloud infrastructure gives enterprises more freedom to build without inheriting more complexity or traditional JVM economics.” “As data management and processing become more central to AI, we're excited to see LakeSail running on Oxide," said Steve Tuck, Co-Founder and CEO of Oxide Computer Company. "Running LakeSail's modern execution engine on Oxide's on-prem Cloud Computer allows customers to gain insights from their data faster while maintaining the control over where that data lives." Read the full press release: lakesail.com/blog/oxide-part…

Curious if your Spark code will actually run on Sail? You can scan it in seconds in your browser or via CLI, and get a full report back as you and your team consider migration. migrate.lakesail.com

Sail ran derived TPC-DS at 3TB and 1TB across a distributed cluster, and the results are the latest evidence for our vision in rethinking distributed compute. Hadoop pioneered open-source distributed processing, built around disk. Spark moved intermediate data into memory, though its shuffle stayed disk-first. When we started Sail, we envisioned another shift in distributed compute, toward doing even more of the work in memory, with an engine built entirely in Rust. We built the first version of Sail's distributed architecture in November 2024, with Arrow Flight carrying shuffle data between workers, and redesigned the cluster control plane earlier this year. Arrow Flight shuffle can look surprising at first. A TPC-DS query with dozens of stages and a few hundred partitions can have more than a million logical shuffle streams active in the cluster. Arrow Flight is built on gRPC and HTTP/2, so one TCP connection per pair of workers carries them all. The same approach has since come up in the DataFusion Comet and Riffle communities as well, and we are glad to have started proving this direction two years ago. The full benchmark ran in 1,289 seconds at 3TB and 444 seconds at 1TB, with no shuffle data written to disk. On similar hardware and settings, Sail outperforms the published TPC-DS results for Spark and Spark accelerators at these scales. The post covers the results, the design, and the full setup: lakesail.com/blog/tpcds-benc…

Sail is the first open-source engine to ship with built-in Jev support! Check out the docs to get started: docs.lakesail.com/sail/lates…