iceberg

DuckLake spec 1 shipped on 13 April 2026 and keeps lake metadata in a SQL database. DuckDB's architecture note puts Iceberg planning at about 100 ms per sequential S3 GET, roughly half a second before a query starts. On AWS, stay on S3 Tables when Athena, Glue, or Redshift must read the table.

For an enterprise lake (~38 TB Iceberg, 8.4k small files), S3 Tables managed compaction cut Athena scan spend −35% — compaction OPEX −90% vs pre–July 2025 pricing on the same workload.

For a specialty retailer (~380 stores, 2.4M loyalty members), S3 Tables Iceberg silver plus Kinesis cart events cut dashboard refresh from 26 hours to 22 minutes — conversion reporting error dropped from 11% to 2%.