Apache Iceberg V4 Spec Moves to Drop Equality Deletes

Apache Iceberg V4 Spec Moves to Drop Equality Deletes

The open lakehouse stack spent this week arguing about what belongs in a spec and what belongs in a release. Iceberg contributors opened a formal push to remove equality deletes from the V4 spec, Parquet closed its vote on a brand new File logical type, and Polaris wrestled with how much consistency its persistence layer owes its users. Underneath the design debates, release trains kept moving: Iceberg Rust, an Iceberg Terraform provider, Arrow JS, Arrow Rust Object Store, DataFusion, Ballista, and Comet all had votes in flight.

By the numbers, the six dev lists carried 268 messages across roughly 80 distinct threads in the past week. Polaris led with 84 messages, Iceberg followed at 71, Parquet posted 51, DataFusion 26, Arrow 20, and the young Ossie project added 16.

Equality Deletes: The Most Consequential Thread

The most consequential thread of the week came from huaxin gao, who restarted the conversation on deprecating equality deletes, this time framed around the V4 spec. Russell Spitzer first proposed the idea back in October 2024, and the blocker then was real: Flink streaming upserts depended on equality deletes and had no practical replacement. The new thread argues that the ground has shifted.

Equality deletes are cheap to write but expensive to read, since every reader must join delete files against candidate rows across a sequence number range. Positional deletes and V3 deletion vectors read far faster, one compact bitmap per data file applied with an O(1) position check. The thread goes past performance too. Equality deletes block CDC and row lineage because the true state of a table requires a full scan while they exist. They also force full rebuilds of materialized views and secondary indexes instead of incremental maintenance.

Flink Conversion Tooling Now Complete

Maximilian Michels reported that the Flink equality delete to deletion vector conversion work is now complete, merged across six commits. The feature ships as a new table maintenance task called ConvertEqualityDeletes, integrated with IcebergSink. Writers keep producing data files and equality deletes as before. The converter reads them, resolves the deletes into deletion vectors using a primary key index backed by RocksDB in Flink state, and commits data files plus DVs to the target branch. Teams can stage writes on a separate branch so readers never see equality deletes at all, or run in-place conversion on a single branch.

Iceberg Rust 0.10.0 RC4

Danny Jones and Shawn Chang called a vote on Iceberg Rust 0.10.0 RC4 after RC3 gathered votes from Kevin Liu, Amogh Jahagirdar, and others earlier in the week. Sung Yun, Renjie Liu, L. C. Hsieh, and Maximilian Michels verified RC4, which includes a check that pyiceberg-core builds and tests cleanly against the release. The Rust implementation now sits underneath the Python ecosystem, so each Rust release carries weight well beyond Rust users. The 0.10.0 line continues the project's steady march toward feature parity with the Java reference implementation.

Iceberg Terraform Provider v0.1.0 RC0

Matt Topol proposed RC0 of the Apache Iceberg Terraform Provider v0.1.0, the project's first release of the provider, with convenience binaries prepared for the Terraform and OpenTofu registries. Neelesh Salian, Alex Stephen, Sung Yun, Talat Uyarer, and Rich Bowen all participated in verification, and an RC1 vote followed as issues surfaced. Once this lands, teams can declare Iceberg resources in the same Terraform plans that manage the rest of their infrastructure.

AI-Assisted Code Review

Gang Wu asked the community about enabling ASF-managed GitHub Copilot code review on Iceberg repositories, starting with iceberg-cpp as a trial. The proposal uses the .asf.yaml setting that ASF Infra now supports, and it follows Apache Arrow, which enabled and tuned the same feature. The thread drew eleven messages from Steve Loughran, Junwang Zhao, Scott Haines, and others, making it the most active Iceberg discussion of the week.

Type System Proposals

Yan Yan opened a discussion on first-class vector type support for Iceberg. Embeddings are everywhere in AI workloads, and today they live in Iceberg as list, which cannot express the invariant that every value shares one dimension. The proposal prefers a dense numeric vector type with compact schema encoding, something like float[768], with fixed dimension, non-null elements, and nullability controlled at the field level.

Meanwhile the collation support discussion between Andrei Tserakhau, Alexander Löser, and Russell Spitzer dug into a genuinely hard question: how much cross-engine interoperability should the format guarantee for string ordering? Alexander laid out the trap in detail. ICU does not keep orderings stable across versions, so two engines on different ICU versions can sort the same strings differently, return different aggregation results, and filter different rows.

Vortex as a File Format?

Martin Prammer proposed adding Vortex as an Iceberg file format, and smartly split the draft into two parts: what criteria any candidate file format should meet, and how Vortex meets them. That framing turns a single-format request into a durable policy, which is exactly what a spec-driven project needs as more formats knock on the door.

Quality and Correctness

Priyadarshini Mitra proposed a ValidateTableIntegrity action that walks the full metadata graph, verifying every referenced file exists on storage. Neelesh Salian, Sung Yun, and Andrei Tserakhau merged their earlier threads into one proposal for shared conformance fixtures, a standalone language-neutral repository modeled on parquet-testing, so every Iceberg implementation checks its reading of the spec against a shared answer key.

Operational Fixes

Oleksii Omhovytskyi asked about release timing for the encrypted deletion vector fix. On natively encrypted format-v3 tables, a merge-on-read UPDATE or DELETE wrote a deletion vector Puffin file without key metadata, and the next read failed. The fix is merged with backports staged on the 1.11.x and 1.10.x branches.

Next Steps

If you use Iceberg in production, watch the V4 deprecation vote and start testing the Flink ConvertEqualityDeletes task. For Terraform users, the provider RC is worth a trial. Rust and Python users should verify their builds against 0.10.0 RC4. The collation and vector type discussions are early, but they signal where the format is heading. Engage on the dev lists if you have opinions.