Package org.apache.beam.sdk.io.iceberg.cdc

Class CdcReadUtils

java.lang.Object
org.apache.beam.sdk.io.iceberg.cdc.CdcReadUtils
public final class CdcReadUtils extends Object
Read-side helpers specific to the CDC source. Keeps ReadUtils focused on the general-purpose append-only read path; everything that takes a SerializableChangelogTask, references DeleteReader, or implements the delete-pushdown row-group skipping lives here.

  • Constructor Details

    • CdcReadUtils

      public CdcReadUtils()

  • Method Details

    • createReader

      public static org.apache.iceberg.io.CloseableIterable<org.apache.iceberg.data.Record> createReader(SerializableChangelogTask task, org.apache.iceberg.Table table, IcebergScanConfig scanConfig, org.apache.iceberg.Schema outputSchema)

    • createReader

      public static org.apache.iceberg.io.CloseableIterable<org.apache.iceberg.data.Record> createReader(SerializableChangelogTask task, org.apache.iceberg.Table table, IcebergScanConfig scanConfig, org.apache.iceberg.Schema outputSchema, org.apache.iceberg.expressions.Expression extraResidual)
      Same as createReader(SerializableChangelogTask, Table, IcebergScanConfig, Schema) but ANDs extraResidual into the task's residual expression. The combined expression is passed to Iceberg's Parquet reader, which uses it as a row-group-level filter (skips row groups whose column statistics cannot match). The caller is still responsible for applying the residual at the row level.

      This is used to push extra predicates (e.g. an equality-delete IN expression) down to the reader for cheap row-group skipping.

    • createReader

      public static org.apache.iceberg.io.CloseableIterable<org.apache.iceberg.data.Record> createReader(SerializableChangelogTask task, org.apache.iceberg.Table table, IcebergScanConfig scanConfig, org.apache.iceberg.Schema outputSchema, org.apache.iceberg.expressions.Expression extraResidual, long start, long length)
      Same as createReader(SerializableChangelogTask, Table, IcebergScanConfig, Schema, Expression) but reads the byte range [start, start + length) of the DataFile. Iceberg's Parquet reader selects the row groups whose starting offset falls within this range, allowing us to prune row-groups by byte-range.

      Callers are responsible for ensuring the requested range stays within the task's assigned range, to avoid reading a section that is meant for another worker.

    • genericDeleteFilter

      public static org.apache.iceberg.data.DeleteFilter<org.apache.iceberg.data.Record> genericDeleteFilter(org.apache.iceberg.Table table, org.apache.iceberg.Schema outputSchema, String dataFilePath, List<SerializableDeleteFile> deletes)
      Returns a filter that skips records marked for deletion.

    • genericDeleteReader

      public static DeleteReader<org.apache.iceberg.data.Record> genericDeleteReader(org.apache.iceberg.Table table, org.apache.iceberg.Schema outputSchema, String dataFilePath, List<SerializableDeleteFile> deletes, DeleteReader.PreloadedDeletes preloadedDeletes)
      Returns a delete reader that reuses delete structures already loaded by CDC planning.

    • changelogRecordsForTask

      public static org.apache.iceberg.io.CloseableIterable<org.apache.iceberg.data.Record> changelogRecordsForTask(SerializableChangelogTask task, org.apache.iceberg.Table table, IcebergScanConfig scanConfig, boolean useProjectedSchema)
      Opens the records that a CDC reader should process for a single SerializableChangelogTask, applying the appropriate delete-filter / delete-reader chain for the task's type:
      • ADDED_ROWS: Collect and return the records that became live in this commit:
        • 1. Iterate over records in the added DataFile
        • 2. Filter out records matched by any added deletes
      • DELETED_ROWS: Return records in the DataFile that are marked for deletion by new DeleteFiles, making sure to first ignore records that have already been marked by previous DeleteFiles:
        • 1. Iterate over records in the referenced DataFile
        • 2. Filter out records matched from existing deletes.
        • 3. Filter out records NOT matched from added deletes
      • DELETED_FILE — every record in the DataFile that wasn't already deleted by existingDeletes.
        • 1. Iterate over records in the referenced DataFile
        • 2. Filter out records matched from existing deletes.

      Projection pushdown should not be used when reading bi-directional tasks because we need to compare all record columns to accurately identify updates. Otherwise, user-configured projection may drop a column that contains real updates. If this happens, the downstream resolver will mistakenly determine the (delete, insert) pair to be a duplicate.

      If CDC metadata columns are requested, this method only adds row-sourced metadata columns (_row_id, _last_updated_sequence_number) to the Iceberg read schema. Changelog context columns are added later by CdcOutputUtils.outputRow(java.util.List<java.lang.String>, org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.io.iceberg.cdc.ChangelogDescriptor, org.apache.beam.sdk.values.ValueKind, org.apache.beam.sdk.values.Row).

    • asParquetInputFile

      public static org.apache.parquet.io.InputFile asParquetInputFile(org.apache.iceberg.io.InputFile icebergFile)
      Adapter from Iceberg's InputFile to Parquet's InputFile, for callers that need to open a Parquet file directly (e.g. to read the footer for row-group pruning decisions). Iceberg has an equivalent internal ParquetIO but it's package-private.