Andy Warfield, an engineer on the S3 team at AWS, published a guest post on All Things Distributed on August 26, 2026, introduced by Werner Vogels. In it, he explains why embedded analytical engines like DuckDB are gaining importance, and announces that DuckLabs, the team that develops DuckDB, is joining AWS.
His reading grid is one of a moving "physics." Where the physical sciences explore invariants, systems computing seeks the elegant trade-off against ratios that shift: memory speed versus network speed, richness of abstractions versus available power. He cites three moments — Berkeley's NOW project, his own work on Xen, and the MonetDB and X100 research at Amsterdam's CWI, where the bottleneck of query processing had shifted from disk to CPU — and notes that these constraints recur in cycles.
Applied to data, this grid explains distributed processing. Processing is always simpler and more efficient on a single fast machine, but when a server's disk or network card can no longer read the desired volume, one partitions. That was the constraint of the early 2000s, the one that produced MapReduce and then Spark's RDDs. Warfield notes two qualities of these systems: they innovated heavily on developer ergonomics, and they accepted a fixed cost of planning and distribution, betting on throughput gained by adding machines rather than on per-unit efficiency.
But the ratios have changed. A current instance offers roughly fifty times the memory, cores, and network bandwidth of the largest EC2 instance from 2007, while dataset growth follows a distribution whose extreme cases form the tail. The 2015 Scalability! But at what COST? paper had already shown that a carefully optimized single-thread implementation could beat distributed frameworks running on one hundred twenty-eight cores.
DuckDB, launched in 2018 by Hannes Mühleisen and Mark Raasveldt, applies this logic: an in-process library analytical engine, running in the application's address space, following SQLite's distribution model. AWS became a DuckLabs customer and then a sponsor of the Iceberg extension, alongside its work on S3 Tables; the extension now supports Iceberg v2 and v3 and exceeds 800,000 downloads per week.
Warfield does not present the embedded model as a replacement: when a job requires a thousand machines, it requires them. What is changing, he writes, is that much of the work done on data never actually needed a cluster. DuckLabs joins AWS as a subsidiary, with the project remaining open source under the MIT license and under the stewardship of the DuckDB Foundation.