org.apache.beam.sdk.io.iceberg

Class IcebergIO.WriteRows

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<Row>,IcebergWriteResult>

org.apache.beam.sdk.io.iceberg.IcebergIO.WriteRows

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

IcebergIO

public abstract static class IcebergIO.WriteRows
extends PTransform<PCollection<Row>,IcebergWriteResult>

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
WriteRows()

Method Summary

Modifier and Type	Method and Description
IcebergWriteResult	expand(PCollection<Row> input) Override this method to specify how this PTransform should be expanded on the given InputT.
IcebergIO.WriteRows	to(DynamicDestinations destinations)
IcebergIO.WriteRows	to(org.apache.iceberg.catalog.TableIdentifier identifier)
IcebergIO.WriteRows	withTriggeringFrequency(Duration triggeringFrequency) Sets the frequency at which data is written to files and a new Snapshot is produced.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

WriteRows

public WriteRows()

Method Detail

to

public IcebergIO.WriteRows to(org.apache.iceberg.catalog.TableIdentifier identifier)

to

public IcebergIO.WriteRows to(DynamicDestinations destinations)

withTriggeringFrequency

public IcebergIO.WriteRows withTriggeringFrequency(Duration triggeringFrequency)

Sets the frequency at which data is written to files and a new Snapshot is produced.

Roughly every triggeringFrequency duration, records are written to data files and appended to the respective table. Each append operation created a new table snapshot.

Generally speaking, increasing this duration will result in fewer, larger data files and fewer snapshots.

This is only applicable when writing an unbounded PCollection (i.e. a streaming pipeline).

expand

public IcebergWriteResult expand(PCollection<Row> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<Row>,IcebergWriteResult>