Package org.apache.beam.sdk.io.csv

Class CsvIO.Write<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<T>,WriteFilesResult<String>>

org.apache.beam.sdk.io.csv.CsvIO.Write<T>

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

CsvIO

public abstract static class CsvIO.Write<T>
extends PTransform<PCollection<T>,WriteFilesResult<String>>
implements HasDisplayData

PTransform for writing CSV files.

See Also:

• Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor	Description
Write()

Method Summary

Modifier and Type	Method	Description
WriteFilesResult<String>	expand(PCollection<T> input)	Override this method to specify how this PTransform should be expanded on the given InputT.
void	populateDisplayData(DisplayData.Builder builder)	Register display data for the given transform or component.
CsvIO.Write<T>	withCompression(Compression compression)	Specifies the Compression of all generated shard files.
CsvIO.Write<T>	withMaxNumWritersPerBundle(@Nullable Integer maxNumWritersPerBundle)	Set the maximum number of writers created in a bundle before spilling to shuffle.
CsvIO.Write<T>	withNoSpilling()	Whether to skip the spilling of data.
CsvIO.Write<T>	withNumShards(Integer numShards)	Specifies to use a given fixed number of shards per window.
CsvIO.Write<T>	withoutSharding()	Forces a single file as output and empty shard name template.
CsvIO.Write<T>	withShardTemplate(String shardTemplate)	Uses the given ShardNameTemplate for naming output files.
CsvIO.Write<T>	withSuffix(String suffix)	Configures the filename suffix for written files.
CsvIO.Write<T>	withTempDirectory(ResourceId tempDirectory)	Set the base directory used to generate temporary files.
CsvIO.Write<T>	withWindowedWrites()	Preserves windowing of input elements and writes them to files based on the element's window.
CsvIO.Write<T>	withWritableByteChannelFactory(FileBasedSink.WritableByteChannelFactory writableByteChannelFactory)	Returns a transform for writing to text files like this one but that has the given FileBasedSink.WritableByteChannelFactory to be used by the FileBasedSink during output.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

Write

public Write()

Method Details

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: PTransform

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class PTransform<PCollection<T>,WriteFilesResult<String>>

Parameters:

builder - The builder to populate with display data.

See Also:

• HasDisplayData

withCompression

public CsvIO.Write<T> withCompression(Compression compression)

Specifies the Compression of all generated shard files.

withNoSpilling

public CsvIO.Write<T> withNoSpilling()

Whether to skip the spilling of data. See WriteFiles.withNoSpilling().

withMaxNumWritersPerBundle

public CsvIO.Write<T> withMaxNumWritersPerBundle(@Nullable Integer maxNumWritersPerBundle)

Set the maximum number of writers created in a bundle before spilling to shuffle. See

invalid reference

WriteFiles#withMaxNumWritersPerBundle()

.

withNumShards

public CsvIO.Write<T> withNumShards(Integer numShards)

Specifies to use a given fixed number of shards per window. See TextIO.Write.withNumShards(int).

withoutSharding

public CsvIO.Write<T> withoutSharding()

Forces a single file as output and empty shard name template. See TextIO.Write.withoutSharding().

withShardTemplate

public CsvIO.Write<T> withShardTemplate(String shardTemplate)

Uses the given ShardNameTemplate for naming output files. See TextIO.Write.withShardNameTemplate(java.lang.String).

withSuffix

public CsvIO.Write<T> withSuffix(String suffix)

Configures the filename suffix for written files. See TextIO.Write.withSuffix(java.lang.String).

withTempDirectory

public CsvIO.Write<T> withTempDirectory(ResourceId tempDirectory)

Set the base directory used to generate temporary files. See TextIO.Write.withTempDirectory(org.apache.beam.sdk.options.ValueProvider<org.apache.beam.sdk.io.fs.ResourceId>).

withWindowedWrites

public CsvIO.Write<T> withWindowedWrites()

Preserves windowing of input elements and writes them to files based on the element's window. See TextIO.Write.withWindowedWrites().

withWritableByteChannelFactory

public CsvIO.Write<T> withWritableByteChannelFactory(FileBasedSink.WritableByteChannelFactory writableByteChannelFactory)

Returns a transform for writing to text files like this one but that has the given FileBasedSink.WritableByteChannelFactory to be used by the FileBasedSink during output. See TextIO.Write.withWritableByteChannelFactory(org.apache.beam.sdk.io.FileBasedSink.WritableByteChannelFactory).

expand

public WriteFilesResult<String> expand(PCollection<T> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<T>,WriteFilesResult<String>>