org.apache.beam.sdk.io

Class TextIO.Write

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<java.lang.String>,PDone>

org.apache.beam.sdk.io.TextIO.Write

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

TextIO

public abstract static class TextIO.Write
extends PTransform<PCollection<java.lang.String>,PDone>

Implementation of TextIO.write().

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<java.lang.String> input) Override this method to specify how this PTransform should be expanded on the given InputT.
protected Coder<java.lang.Void>	getDefaultOutputCoder() Returns the default Coder to use for the output of this single-output PTransform.
void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
TextIO.Write	to(ResourceId filenamePrefix) Writes to text files with prefix from the given resource.
TextIO.Write	to(java.lang.String filenamePrefix) Writes to text files with the given prefix.
TextIO.Write	to(ValueProvider<java.lang.String> outputPrefix) Like to(String).
TextIO.Write	toResource(ValueProvider<ResourceId> filenamePrefix) Like to(ResourceId).
TextIO.Write	withFilenamePolicy(FileBasedSink.FilenamePolicy filenamePolicy) Configures the FileBasedSink.FilenamePolicy that will be used to name written files.
TextIO.Write	withFooter(java.lang.String footer) Adds a footer string to each file.
TextIO.Write	withHeader(java.lang.String header) Adds a header string to each file.
TextIO.Write	withNumShards(int numShards) Configures the number of output shards produced overall (when using unwindowed writes) or per-window (when using windowed writes).
TextIO.Write	withoutSharding() Forces a single file as output and empty shard name template.
TextIO.Write	withShardNameTemplate(java.lang.String shardTemplate) Uses the given ShardNameTemplate for naming output files.
TextIO.Write	withSuffix(java.lang.String filenameSuffix) Configures the filename suffix for written files.
TextIO.Write	withWindowedWrites()
TextIO.Write	withWritableByteChannelFactory(FileBasedSink.WritableByteChannelFactory writableByteChannelFactory) Returns a transform for writing to text files like this one but that has the given FileBasedSink.WritableByteChannelFactory to be used by the FileBasedSink during output.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

to

public TextIO.Write to(java.lang.String filenamePrefix)

Writes to text files with the given prefix. The given prefix can reference any FileSystem on the classpath.

The name of the output files will be determined by the FileBasedSink.FilenamePolicy used.

By default, a DefaultFilenamePolicy will be used built using the specified prefix to define the base output directory and file prefix, a shard identifier (see withNumShards(int)), and a common suffix (if supplied using withSuffix(String)).

This default policy can be overridden using #withFilenamePolicy(FilenamePolicy), in which case withShardNameTemplate(String) and withSuffix(String) should not be set.

to

@Experimental(value=FILESYSTEM)
public TextIO.Write to(ResourceId filenamePrefix)

Writes to text files with prefix from the given resource.

The name of the output files will be determined by the FileBasedSink.FilenamePolicy used.

By default, a DefaultFilenamePolicy will be used built using the specified prefix to define the base output directory and file prefix, a shard identifier (see withNumShards(int)), and a common suffix (if supplied using withSuffix(String)).

This default policy can be overridden using #withFilenamePolicy(FilenamePolicy), in which case withShardNameTemplate(String) and withSuffix(String) should not be set.

to

public TextIO.Write to(ValueProvider<java.lang.String> outputPrefix)

Like to(String).

toResource

@Experimental(value=FILESYSTEM)
public TextIO.Write toResource(ValueProvider<ResourceId> filenamePrefix)

Like to(ResourceId).

withShardNameTemplate

public TextIO.Write withShardNameTemplate(java.lang.String shardTemplate)

Uses the given ShardNameTemplate for naming output files. This option may only be used when #withFilenamePolicy(FilenamePolicy) has not been configured.

See DefaultFilenamePolicy for how the prefix, shard name template, and suffix are used.

withSuffix

public TextIO.Write withSuffix(java.lang.String filenameSuffix)

Configures the filename suffix for written files. This option may only be used when #withFilenamePolicy(FilenamePolicy) has not been configured.

See DefaultFilenamePolicy for how the prefix, shard name template, and suffix are used.

withFilenamePolicy

public TextIO.Write withFilenamePolicy(FileBasedSink.FilenamePolicy filenamePolicy)

Configures the FileBasedSink.FilenamePolicy that will be used to name written files.

withNumShards

public TextIO.Write withNumShards(int numShards)

Configures the number of output shards produced overall (when using unwindowed writes) or per-window (when using windowed writes).

For unwindowed writes, constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

Parameters:

numShards - the number of shards to use, or 0 to let the system decide.

withoutSharding

public TextIO.Write withoutSharding()

Forces a single file as output and empty shard name template. This option is only compatible with unwindowed writes.

For unwindowed writes, constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

This is equivalent to .withNumShards(1).withShardNameTemplate("")

withHeader

public TextIO.Write withHeader(@Nullable
                               java.lang.String header)

Adds a header string to each file. A newline after the header is added automatically.

A null value will clear any previously configured header.

withFooter

public TextIO.Write withFooter(@Nullable
                               java.lang.String footer)

Adds a footer string to each file. A newline after the footer is added automatically.

A null value will clear any previously configured footer.

withWritableByteChannelFactory

public TextIO.Write withWritableByteChannelFactory(FileBasedSink.WritableByteChannelFactory writableByteChannelFactory)

Returns a transform for writing to text files like this one but that has the given FileBasedSink.WritableByteChannelFactory to be used by the FileBasedSink during output. The default is value is FileBasedSink.CompressionType.UNCOMPRESSED.

A null value will reset the value to the default value mentioned above.

withWindowedWrites

public TextIO.Write withWindowedWrites()

expand

public PDone expand(PCollection<java.lang.String> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<java.lang.String>,PDone>

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: PTransform

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class PTransform<PCollection<java.lang.String>,PDone>

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData

getDefaultOutputCoder

protected Coder<java.lang.Void> getDefaultOutputCoder()

Description copied from class: PTransform

Returns the default Coder to use for the output of this single-output PTransform.

By default, always throws

Overrides:

getDefaultOutputCoder in class PTransform<PCollection<java.lang.String>,PDone>