The iceberg-cpp library is a C++ implementation of the Apache Iceberg table format specification. It provides a native C++ API for reading, writing, and managing Iceberg tables with support for ACID transactions, schema evolution, time travel queries, and multiple storage backends.
This document provides a high-level overview of the library's architecture, major components, and design principles. For detailed information on specific topics:
Sources: src/iceberg/table.h src/iceberg/table_metadata.h README.md
The library implements the following Iceberg capabilities:
| Feature | Description | Key Components |
|---|---|---|
| ACID Transactions | Optimistic concurrency control with atomic commits | Transaction, TableMetadataBuilder, TableRequirement |
| Schema Evolution | Add, drop, rename, and reorder columns without rewriting data | UpdateSchema, Schema, SchemaField |
| Time Travel | Query historical table snapshots | Snapshot, SnapshotRef, TableScan |
| Partition Evolution | Change partitioning scheme without data migration | PartitionSpec, UpdatePartitionSpec |
| Multiple File Formats | Read and write Avro, Parquet, and ORC files | Reader, Writer, ReaderFactory |
| Predicate Pushdown | Filter data at partition, file, and row levels | Expression, ManifestEvaluator, InclusiveMetricsEvaluator |
| Hidden Partitioning | Partition transforms applied transparently | Transform, PartitionField |
| Sort Orders | Specify and maintain data ordering | SortOrder, SortField |
| Delete Files | Position and equality deletes for row-level operations | DeleteFileIndex, ManifestEntry |
| REST Catalog | Remote catalog protocol implementation | RestCatalog, AuthManager |
Sources: src/iceberg/table.h37-169 src/iceberg/transaction.h32-151 src/iceberg/table_metadata.h64-165
The library follows a four-layer architecture that separates concerns and enables modularity:
Figure 1: Four-Layer Architecture
Sources: src/iceberg/table.h src/iceberg/table_metadata.h src/iceberg/table_scan.h src/iceberg/file_io.h
The following diagram shows how major components interact during typical operations:
Figure 2: Component Interaction Graph
Sources: src/iceberg/table.h38-182 src/iceberg/transaction.h33-151 src/iceberg/table_metadata.h220-488 src/iceberg/update/pending_update.h35-94
The Table class is the primary entry point for interacting with Iceberg tables. It provides access to table metadata and factory methods for creating operations:
schema(), spec(), sort_order(): Access current table configurationNewScan(): Create query buildersNewUpdateSchema(), NewFastAppend(), etc.: Create update operationsRefresh(): Reload metadata from catalogThe Catalog interface abstracts table discovery and storage:
RestCatalog for remote catalogs, InMemoryCatalog for testingLoadTable(), CreateTable(), UpdateTable(), namespace managementSources: src/iceberg/table.h38-224 src/iceberg/catalog.h
The metadata layer manages versioned table state through immutable snapshots:
The TableMetadata struct (src/iceberg/table_metadata.h72-165) contains all table configuration:
Sources: src/iceberg/table_metadata.h72-165 src/iceberg/table_metadata.cc1-1679
The TableMetadataBuilder class (src/iceberg/table_metadata.h220-488) provides a fluent API for constructing new metadata versions:
TableUpdate instancesTableRequirement instances for optimistic concurrency controlSources: src/iceberg/table_metadata.h220-488 src/iceberg/table_metadata.cc543-1679
The Transaction class (src/iceberg/transaction.h33-151) coordinates atomic commits of multiple table changes:
Figure 3: Transaction Flow for Schema Update
Transaction modes:
Sources: src/iceberg/transaction.h33-151 src/iceberg/transaction.cc1-432
The type system is built on a hierarchy of Type classes:
Figure 4: Type System Hierarchy
A Schema (src/iceberg/schema.h49-198) is a StructType with additional metadata:
schema_id for tracking schema evolutionSources: src/iceberg/type.h44-362 src/iceberg/type.cc1-439 src/iceberg/schema.h49-198 src/iceberg/schema.cc1-307
The query path transforms high-level table scans into executable file-level tasks:
Figure 5: Query Execution Pipeline
Key optimizations:
Sources: src/iceberg/table_scan.h src/iceberg/manifest/manifest_group.h src/iceberg/file_reader.h
The library supports multiple file formats through a plugin architecture:
Figure 6: File Format Architecture
Format implementations are registered at startup (src/iceberg/avro/avro_register.cc src/iceberg/parquet/parquet_register.cc) and selected based on file extension or explicit configuration.
Sources: src/iceberg/file_reader.h src/iceberg/file_writer.h src/iceberg/avro/avro_reader.h src/iceberg/parquet/parquet_reader.h
A Snapshot (src/iceberg/snapshot.h) represents an immutable point-in-time view of a table:
snapshot_id and sequence_numberSnapshotRefTime travel is achieved by:
TableScanSources: src/iceberg/snapshot.h src/iceberg/table_scan.h
A PartitionSpec (src/iceberg/partition_spec.h) defines how data is organized:
Partition evolution allows changing specs without rewriting data:
spec_idspec_idSources: src/iceberg/partition_spec.h src/iceberg/transform.h
The manifest system tracks data file locations and statistics:
Figure 7: Manifest Structure
Each ManifestEntry contains:
Sources: src/iceberg/manifest/manifest_reader.h src/iceberg/manifest/manifest_list.h
The library uses a Result<T> monad (src/iceberg/result.h) for functional error handling:
Error categories are defined in ErrorKind:
kNotFound: Resource doesn't existkInvalidArgument: Bad input parameterskInvalidSchema: Schema validation failurekCommitFailed: Optimistic concurrency conflictkValidationFailed: Business rule violationSources: src/iceberg/result.h src/iceberg/exception.h
Core data structures are immutable once constructed:
TableMetadata: Modified through TableMetadataBuilderSchema: Created through factory methods or builderSnapshot: Never modified after creationThis design enables:
Sources: src/iceberg/table_metadata.h220-488 src/iceberg/schema.h43-68
The library uses Apache Arrow's C Data Interface for zero-copy data exchange:
ArrowArrayStream: Streaming data interfaceArrowSchema: Schema descriptionArrowArray: Columnar batch dataThis enables interoperability with Arrow, DuckDB, Polars, and other systems without serialization overhead.
Sources: src/iceberg/arrow_c_data.h src/iceberg/file_reader.h
The library supports both CMake and Meson build systems with modular output:
| Library | Contents | Dependencies |
|---|---|---|
libiceberg | Core metadata and planning | nanoarrow, nlohmann-json, CRoaring, zlib |
libiceberg_bundle | File format support (optional) | Arrow, Avro, Parquet, Zstd |
libiceberg_rest | REST catalog (optional) | cpr, libcurl |
Build flags control which features are included:
ICEBERG_BUILD_BUNDLE: Enable Avro/Parquet supportICEBERG_BUILD_REST: Enable REST catalog clientICEBERG_BUILD_TESTS: Build test suiteSources: src/iceberg/CMakeLists.txt1-250 src/iceberg/meson.build1-230 cmake_modules/IcebergThirdpartyToolchain.cmake
For detailed information on specific aspects of the library:
Refresh this wiki
This wiki was recently refreshed. Please wait 7 days to refresh again.