Package org.apache.beam.sdk.io.iceberg

Class IcebergWriteResult

java.lang.Object

org.apache.beam.sdk.io.iceberg.IcebergWriteResult

All Implemented Interfaces:

POutput

public final class IcebergWriteResult
extends Object
implements POutput

The output of an IcebergIO write: the snapshots each destination table committed, plus the two diversion outputs the CDC sink can produce.

Only getSnapshots() is always present. getDeadLetterRows() (late-but-valid records) and getFailedRows() (per-record poison rows) are non-null only for results built by cdc(org.apache.beam.sdk.Pipeline, org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String, org.apache.beam.sdk.io.iceberg.SnapshotInfo>>, org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.Row>), that is, by IcebergIO.writeCdcRows, and getFailedRows() additionally only when error handling was enabled. The append-only sink ( IcebergIO.writeRows) leaves both null rather than exposing outputs that can never carry data.

Method Summary

Modifier and Type	Method	Description
static IcebergWriteResult	cdc(Pipeline pipeline, PCollection<KV<String,SnapshotInfo>> snapshots, PCollection<Row> deadLetterRows, @Nullable PCollection<Row> failedRows)	Returns an IcebergWriteResult for the CDC sink, exposing the committed-snapshot snapshots, the replayable deadLetterRows (see getDeadLetterRows()), and the optional per-record failedRows (see getFailedRows(), null when error handling is off).
Map<TupleTag<?>,PValue>	expand()	Expands this POutput into a list of its component output PValues.
void	finishSpecifyingOutput(String transformName, PInput input, PTransform<?,?> transform)	As part of applying the producing PTransform, finalizes this output to make it ready for being used as an input and for running.
PCollection<Row>	getDeadLetterRows()	The replayable dead-letter Rows from the CDC sink: records whose grouped pane fired late, i.e.
PCollection<Row>	getFailedRows()	The per-record poison rows diverted by the CDC sink when error handling is enabled (see WriteCdcRows.withErrorHandling); schema is failed_row ROW + error_message STRING (see ErrorHandling).
Pipeline	getPipeline()	Returns the owning Pipeline of this POutput.
PCollection<KV<String,SnapshotInfo>>	getSnapshots()	The committed snapshots, keyed by destination.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

getSnapshots

public PCollection<KV<String,SnapshotInfo>> getSnapshots()

The committed snapshots, keyed by destination. A window committed in a commit fire that later fails may not re-emit its SnapshotInfo on the retry (table state is unaffected).

getDeadLetterRows

public PCollection<Row> getDeadLetterRows()

The replayable dead-letter Rows from the CDC sink: records whose grouped pane fired late, i.e. after the watermark had passed their commit window's end. Schema is record ROW<dataSchema> + change_type STRING + sequence_number INT64 + destination STRING. To replay, unnest record, map change_type/sequence_number as the sink's control columns, and route each row by destination. Replaying is only safe while no newer change for those keys has committed; a stale replay's equality delete removes the newer row.

Returns:

the dead-letter PCollection<Row> for results produced by IcebergIO.writeCdcRows/cdc(org.apache.beam.sdk.Pipeline, org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String, org.apache.beam.sdk.io.iceberg.SnapshotInfo>>, org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.Row>). Throws when attempting to retrieve it for the append-only sink (IcebergIO.writeRows) as has no dead-letter output.

getFailedRows

public PCollection<Row> getFailedRows()

The per-record poison rows diverted by the CDC sink when error handling is enabled (see WriteCdcRows.withErrorHandling); schema is failed_row ROW + error_message STRING (see ErrorHandling). Distinct from getDeadLetterRows() (which carries late-but-valid records).

Returns:

the failed-rows PCollection<Row> if error handling was enabled for the CDC sink. Will throw otherwise.

cdc

@Internal
public static IcebergWriteResult cdc(Pipeline pipeline,
 PCollection<KV<String,SnapshotInfo>> snapshots,
 PCollection<Row> deadLetterRows,
 @Nullable PCollection<Row> failedRows)

Returns an IcebergWriteResult for the CDC sink, exposing the committed-snapshot snapshots, the replayable deadLetterRows (see getDeadLetterRows()), and the optional per-record failedRows (see getFailedRows(), null when error handling is off).

getPipeline

public Pipeline getPipeline()

Description copied from interface: POutput

Returns the owning Pipeline of this POutput.

Specified by:

getPipeline in interface POutput

expand

public Map<TupleTag<?>,PValue> expand()

Description copied from interface: POutput

Expands this POutput into a list of its component output PValues.

• A PValue expands to itself.
• A tuple or list of PValues (such as PCollectionTuple or PCollectionList) expands to its component PValue PValues.

Not intended to be invoked directly by user code.

Specified by:

expand in interface POutput

finishSpecifyingOutput

public void finishSpecifyingOutput(String transformName,
 PInput input,
 PTransform<?,?> transform)

Description copied from interface: POutput

As part of applying the producing PTransform, finalizes this output to make it ready for being used as an input and for running.

This includes ensuring that all PCollections have Coders specified or defaulted.

Automatically invoked whenever this POutput is output, after POutput.finishSpecifyingOutput(String, PInput, PTransform) has been called on each component PValue returned by POutput.expand().

Specified by:

finishSpecifyingOutput in interface POutput