Package org.apache.beam.sdk.io

Class FileIO.Write<DestinationT,UserT>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>

org.apache.beam.sdk.io.FileIO.Write<DestinationT,UserT>

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

FileIO

public abstract static class FileIO.Write<DestinationT,UserT>
extends PTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>

Implementation of FileIO.write() and FileIO.writeDynamic().

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	FileIO.Write.FileNaming	A policy for generating names for shard files.

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor	Description
Write()

Method Summary

Modifier and Type	Method	Description
FileIO.Write<DestinationT,UserT>	by(Contextful<Contextful.Fn<UserT,DestinationT>> destinationFn)	Like by(org.apache.beam.sdk.transforms.SerializableFunction<UserT, DestinationT>), but with access to context such as side inputs.
FileIO.Write<DestinationT,UserT>	by(SerializableFunction<UserT,DestinationT> destinationFn)	Specifies how to partition elements into groups ("destinations").
static FileIO.Write.FileNaming	defaultNaming(String prefix, String suffix)
static FileIO.Write.FileNaming	defaultNaming(ValueProvider<String> prefix, ValueProvider<String> suffix)	Defines a default FileIO.Write.FileNaming which will use the prefix and suffix supplied to create a name based on the window, pane, number of shards, shard index, and compression.
WriteFilesResult<DestinationT>	expand(PCollection<UserT> input)	Override this method to specify how this PTransform should be expanded on the given InputT.
static FileIO.Write.FileNaming	relativeFileNaming(ValueProvider<String> baseDirectory, FileIO.Write.FileNaming innerNaming)
FileIO.Write<DestinationT,UserT>	to(String directory)	Specifies a common directory for all generated files.
FileIO.Write<DestinationT,UserT>	to(ValueProvider<String> directory)	Like to(String) but with a ValueProvider.
FileIO.Write<DestinationT,UserT>	via(FileIO.Sink<UserT> sink)	Like via(Contextful), but uses the same FileIO.Sink for all destinations.
FileIO.Write<DestinationT,UserT>	via(Contextful<Contextful.Fn<DestinationT,FileIO.Sink<UserT>>> sinkFn)	Like via(Contextful, Contextful), but the output type of the sink is the same as the type of the input collection.
<OutputT> FileIO.Write<DestinationT,UserT>	via(Contextful<Contextful.Fn<UserT,OutputT>> outputFn, FileIO.Sink<OutputT> sink)	Like via(Contextful, Contextful), but uses the same sink for all destinations.
<OutputT> FileIO.Write<DestinationT,UserT>	via(Contextful<Contextful.Fn<UserT,OutputT>> outputFn, Contextful<Contextful.Fn<DestinationT,FileIO.Sink<OutputT>>> sinkFn)	Specifies how to create a FileIO.Sink for a particular destination and how to map the element type to the sink's output type.
FileIO.Write<DestinationT,UserT>	withAutoSharding()
FileIO.Write<DestinationT,UserT>	withBadRecordErrorHandler(ErrorHandler<BadRecord,?> errorHandler)	Configures a new FileIO.Write with an ErrorHandler.
FileIO.Write<DestinationT,UserT>	withCompression(Compression compression)	Specifies to compress all generated shard files using the given Compression and, by default, append the respective extension to the filename.
FileIO.Write<DestinationT,UserT>	withDestinationCoder(Coder<DestinationT> destinationCoder)	Specifies a Coder for the destination type, if it can not be inferred from by(org.apache.beam.sdk.transforms.SerializableFunction<UserT, DestinationT>).
FileIO.Write<DestinationT,UserT>	withEmptyGlobalWindowDestination(DestinationT emptyWindowDestination)	If withIgnoreWindowing() is specified, specifies a destination to be used in case the collection is empty, to generate the (only, empty) output file.
FileIO.Write<DestinationT,UserT>	withIgnoreWindowing()	Deprecated. Avoid usage of this method: its effects are complex and it will be removed in future versions of Beam.
FileIO.Write<DestinationT,UserT>	withMaxNumWritersPerBundle(@Nullable Integer maxNumWritersPerBundle)	Set the maximum number of writers created in a bundle before spilling to shuffle.
FileIO.Write<DestinationT,UserT>	withNaming(FileIO.Write.FileNaming naming)	Specifies a custom strategy for generating filenames.
FileIO.Write<DestinationT,UserT>	withNaming(Contextful<Contextful.Fn<DestinationT,FileIO.Write.FileNaming>> namingFn)	Like withNaming(SerializableFunction) but allows accessing context, such as side inputs, from the function.
FileIO.Write<DestinationT,UserT>	withNaming(SerializableFunction<DestinationT,FileIO.Write.FileNaming> namingFn)	Specifies a custom strategy for generating filenames depending on the destination, similar to withNaming(FileNaming).
FileIO.Write<DestinationT,UserT>	withNoSpilling()	See WriteFiles.withNoSpilling().
FileIO.Write<DestinationT,UserT>	withNumShards(int numShards)	Specifies to use a given fixed number of shards per window.
FileIO.Write<DestinationT,UserT>	withNumShards(@Nullable ValueProvider<Integer> numShards)	Like withNumShards(int).
FileIO.Write<DestinationT,UserT>	withPrefix(String prefix)	Specifies a common prefix to use for all generated filenames, if using the default file naming.
FileIO.Write<DestinationT,UserT>	withPrefix(ValueProvider<String> prefix)	Like withPrefix(String) but with a ValueProvider.
FileIO.Write<DestinationT,UserT>	withSharding(PTransform<PCollection<UserT>,PCollectionView<Integer>> sharding)	Specifies a PTransform to use for computing the desired number of shards in each window.
FileIO.Write<DestinationT,UserT>	withSuffix(String suffix)	Specifies a common suffix to use for all generated filenames, if using the default file naming.
FileIO.Write<DestinationT,UserT>	withSuffix(ValueProvider<String> suffix)	Like withSuffix(String) but with a ValueProvider.
FileIO.Write<DestinationT,UserT>	withTempDirectory(String tempDirectory)	Specifies a directory into which all temporary files will be placed.
FileIO.Write<DestinationT,UserT>	withTempDirectory(ValueProvider<String> tempDirectory)	Like withTempDirectory(String).

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

Write

public Write()

Method Details

defaultNaming

public static FileIO.Write.FileNaming defaultNaming(String prefix,
 String suffix)

defaultNaming

public static FileIO.Write.FileNaming defaultNaming(ValueProvider<String> prefix,
 ValueProvider<String> suffix)

Defines a default FileIO.Write.FileNaming which will use the prefix and suffix supplied to create a name based on the window, pane, number of shards, shard index, and compression. Removes window when in the GlobalWindow and pane info when it is the only firing of the pane.

relativeFileNaming

public static FileIO.Write.FileNaming relativeFileNaming(ValueProvider<String> baseDirectory,
 FileIO.Write.FileNaming innerNaming)

by

public FileIO.Write<DestinationT,UserT> by(SerializableFunction<UserT,DestinationT> destinationFn)

Specifies how to partition elements into groups ("destinations").

by

public FileIO.Write<DestinationT,UserT> by(Contextful<Contextful.Fn<UserT,DestinationT>> destinationFn)

Like by(org.apache.beam.sdk.transforms.SerializableFunction<UserT, DestinationT>), but with access to context such as side inputs.

via

public <OutputT> FileIO.Write<DestinationT,UserT> via(Contextful<Contextful.Fn<UserT,OutputT>> outputFn,
 Contextful<Contextful.Fn<DestinationT,FileIO.Sink<OutputT>>> sinkFn)

Specifies how to create a FileIO.Sink for a particular destination and how to map the element type to the sink's output type. The sink function must create a new FileIO.Sink instance every time it is called.

via

public <OutputT> FileIO.Write<DestinationT,UserT> via(Contextful<Contextful.Fn<UserT,OutputT>> outputFn,
 FileIO.Sink<OutputT> sink)

Like via(Contextful, Contextful), but uses the same sink for all destinations.

via

public FileIO.Write<DestinationT,UserT> via(Contextful<Contextful.Fn<DestinationT,FileIO.Sink<UserT>>> sinkFn)

Like via(Contextful, Contextful), but the output type of the sink is the same as the type of the input collection. The sink function must create a new FileIO.Sink instance every time it is called.

via

public FileIO.Write<DestinationT,UserT> via(FileIO.Sink<UserT> sink)

Like via(Contextful), but uses the same FileIO.Sink for all destinations.

to

public FileIO.Write<DestinationT,UserT> to(String directory)

Specifies a common directory for all generated files. A temporary generated sub-directory of this directory will be used as the temp directory, unless overridden by withTempDirectory(java.lang.String).

to

public FileIO.Write<DestinationT,UserT> to(ValueProvider<String> directory)

Like to(String) but with a ValueProvider.

withPrefix

public FileIO.Write<DestinationT,UserT> withPrefix(String prefix)

Specifies a common prefix to use for all generated filenames, if using the default file naming. Incompatible with withNaming(org.apache.beam.sdk.io.FileIO.Write.FileNaming).

withPrefix

public FileIO.Write<DestinationT,UserT> withPrefix(ValueProvider<String> prefix)

Like withPrefix(String) but with a ValueProvider.

withSuffix

public FileIO.Write<DestinationT,UserT> withSuffix(String suffix)

Specifies a common suffix to use for all generated filenames, if using the default file naming. Incompatible with withNaming(org.apache.beam.sdk.io.FileIO.Write.FileNaming).

withSuffix

public FileIO.Write<DestinationT,UserT> withSuffix(ValueProvider<String> suffix)

Like withSuffix(String) but with a ValueProvider.

withNaming

public FileIO.Write<DestinationT,UserT> withNaming(FileIO.Write.FileNaming naming)

Specifies a custom strategy for generating filenames. All generated filenames will be resolved relative to the directory specified in to(java.lang.String), if any.

Incompatible with withSuffix(java.lang.String).

This can only be used in combination with FileIO.write() but not FileIO.writeDynamic().

withNaming

public FileIO.Write<DestinationT,UserT> withNaming(SerializableFunction<DestinationT,FileIO.Write.FileNaming> namingFn)

Specifies a custom strategy for generating filenames depending on the destination, similar to withNaming(FileNaming).

This can only be used in combination with FileIO.writeDynamic() but not FileIO.write().

withNaming

public FileIO.Write<DestinationT,UserT> withNaming(Contextful<Contextful.Fn<DestinationT,FileIO.Write.FileNaming>> namingFn)

Like withNaming(SerializableFunction) but allows accessing context, such as side inputs, from the function.

withTempDirectory

public FileIO.Write<DestinationT,UserT> withTempDirectory(String tempDirectory)

Specifies a directory into which all temporary files will be placed.

withTempDirectory

public FileIO.Write<DestinationT,UserT> withTempDirectory(ValueProvider<String> tempDirectory)

Like withTempDirectory(String).

withCompression

public FileIO.Write<DestinationT,UserT> withCompression(Compression compression)

Specifies to compress all generated shard files using the given Compression and, by default, append the respective extension to the filename.

withEmptyGlobalWindowDestination

public FileIO.Write<DestinationT,UserT> withEmptyGlobalWindowDestination(DestinationT emptyWindowDestination)

If withIgnoreWindowing() is specified, specifies a destination to be used in case the collection is empty, to generate the (only, empty) output file.

withDestinationCoder

public FileIO.Write<DestinationT,UserT> withDestinationCoder(Coder<DestinationT> destinationCoder)

Specifies a Coder for the destination type, if it can not be inferred from by(org.apache.beam.sdk.transforms.SerializableFunction<UserT, DestinationT>).

withNumShards

public FileIO.Write<DestinationT,UserT> withNumShards(int numShards)

Specifies to use a given fixed number of shards per window. 0 means runner-determined sharding. Specifying a non-zero value may hurt performance, because it will limit the parallelism of writing and will introduce an extra GroupByKey operation.

withNumShards

public FileIO.Write<DestinationT,UserT> withNumShards(@Nullable ValueProvider<Integer> numShards)

Like withNumShards(int). Specifying null means runner-determined sharding.

withSharding

public FileIO.Write<DestinationT,UserT> withSharding(PTransform<PCollection<UserT>,PCollectionView<Integer>> sharding)

Specifies a PTransform to use for computing the desired number of shards in each window.

withIgnoreWindowing

@Deprecated
public FileIO.Write<DestinationT,UserT> withIgnoreWindowing()

Deprecated.

Avoid usage of this method: its effects are complex and it will be removed in future versions of Beam. Right now it exists for compatibility with WriteFiles.

Specifies to ignore windowing information in the input, and instead rewindow it to global window with the default trigger.

withAutoSharding

public FileIO.Write<DestinationT,UserT> withAutoSharding()

withNoSpilling

public FileIO.Write<DestinationT,UserT> withNoSpilling()

See WriteFiles.withNoSpilling().

withMaxNumWritersPerBundle

public FileIO.Write<DestinationT,UserT> withMaxNumWritersPerBundle(@Nullable Integer maxNumWritersPerBundle)

Set the maximum number of writers created in a bundle before spilling to shuffle. See

invalid reference

WriteFiles#withMaxNumWritersPerBundle()

.

withBadRecordErrorHandler

public FileIO.Write<DestinationT,UserT> withBadRecordErrorHandler(ErrorHandler<BadRecord,?> errorHandler)

Configures a new FileIO.Write with an ErrorHandler. For configuring an ErrorHandler, see ErrorHandler. Whenever a record is formatted, or a lookup for a dynamic destination is performed, and that operation fails, the exception is passed to the error handler. This is intended to handle any errors related to the data of a record, but not any connectivity or IO errors related to the literal writing of a record.

expand

public WriteFilesResult<DestinationT> expand(PCollection<UserT> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>