org.apache.beam.sdk.io.hadoop.format

Interface HadoopFormatIO.Write.WriteBuilder<KeyT,ValueT>

Type Parameters:

KeyT - Key type to write

ValueT - Value type to write

Enclosing class:

HadoopFormatIO.Write<KeyT,ValueT>

public static interface HadoopFormatIO.Write.WriteBuilder<KeyT,ValueT>

Main builder of Write transformation.

Method Summary

Modifier and Type	Method and Description
HadoopFormatIO.Write.PartitionedWriterBuilder<KeyT,ValueT>	withConfiguration(org.apache.hadoop.conf.Configuration config) Writes to the sink using the options provided by the given hadoop configuration.
HadoopFormatIO.Write.ExternalSynchronizationBuilder<KeyT,ValueT>	withConfigurationTransform(PTransform<PCollection<? extends KV<KeyT,ValueT>>,PCollectionView<org.apache.hadoop.conf.Configuration>> configTransform) Writes to the sink using configuration created by provided configurationTransformation.

Method Detail

withConfiguration

HadoopFormatIO.Write.PartitionedWriterBuilder<KeyT,ValueT> withConfiguration(org.apache.hadoop.conf.Configuration config)

Writes to the sink using the options provided by the given hadoop configuration.

Note: Works only for PCollection.IsBounded.BOUNDED PCollection with global WindowingStrategy.

Parameters:

config - hadoop configuration.

Returns:

WriteBuilder with set configuration

Throws:

java.lang.NullPointerException - when the configuration is null

See Also:

for required hadoop {@link Configuration} properties

withConfigurationTransform

HadoopFormatIO.Write.ExternalSynchronizationBuilder<KeyT,ValueT> withConfigurationTransform(PTransform<PCollection<? extends KV<KeyT,ValueT>>,PCollectionView<org.apache.hadoop.conf.Configuration>> configTransform)

Writes to the sink using configuration created by provided configurationTransformation.

This type is useful especially for processing unbounded windowed data but can be used also for batch processing.

Supports only PCollection with DefaultTriggering and without allowed lateness

Parameters:

configTransform - configuration transformation interface

Returns:

WriteBuilder with set configuration transformation

Throws:

java.lang.NullPointerException - when configurationTransformation is null

See Also:

for required hadoop {@link Configuration} properties