Package org.apache.beam.sdk.io.iceberg.cdc

Class ReadFromChangelogs

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollectionTuple,ReadFromChangelogs.Output>

org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs

All Implemented Interfaces:

Serializable, HasDisplayData

public class ReadFromChangelogs
extends PTransform<PCollectionTuple,ReadFromChangelogs.Output>

A PTransform that processes batches of ChangelogScanTasks and routes them accordingly:

• Records from Uni-directional batches are directly emitted, as INSERT or DELETE kind
• Records from Bi-directional batches are compared against the Primary Key overlap range:
  • if outside the overlap, emit directly as INSERT or DELETE kind
  • if inside the overlap, key by (snapshot seq#, pk) and route to downstream CoGroupByKey and final resolution by ResolveChanges

We first key bi-directional rows by (snapshot sequence number, primary key) before sending to CoGroupByKey to ensure they stay isolated from other PKs or snapshots. Inserts are routed to

A ChangelogScanTask comes in three types:

1. AddedRowsScanTask: Indicates records have been inserted by a new DataFile.
2. DeletedRowsScanTask: Indicates records have been deleted using a DeleteFile.
3. DeletedDataFileScanTask: Indicates a whole DataFile has been deleted.

Each of these types need to be processed differently. More details in CdcReadUtils.changelogRecordsForTask(org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask, org.apache.iceberg.Table, org.apache.beam.sdk.io.iceberg.IcebergScanConfig, boolean).

CDC metadata has two entry points in this transform. Row metadata columns are requested from the Iceberg reader by CdcReadUtils and travel inside intermediate rows until final output assembly. Snapshot metadata columns come from the ChangelogDescriptor / CdcRowDescriptor carried with each task or shuffled row, and _change_type comes from the emitted change kind. Final user-visible rows are assembled by CdcOutputUtils.outputRow(java.util.List<java.lang.String>, org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.io.iceberg.cdc.ChangelogDescriptor, org.apache.beam.sdk.values.ValueKind, org.apache.beam.sdk.values.Row), which appends all requested metadata as top-level columns in the configured order.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ReadFromChangelogs.Output

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Method Summary

Modifier and Type	Method	Description
ReadFromChangelogs.Output	expand(PCollectionTuple input)	Override this method to specify how this PTransform should be expanded on the given InputT.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Details

expand

public ReadFromChangelogs.Output expand(PCollectionTuple input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollectionTuple,ReadFromChangelogs.Output>