org.apache.beam.sdk.io

Class AvroIO.Write<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<T>,PDone>

org.apache.beam.sdk.io.AvroIO.Write<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

AvroIO

public abstract static class AvroIO.Write<T>
extends PTransform<PCollection<T>,PDone>

Implementation of AvroIO.write(java.lang.Class<T>).

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<T> input) Applies this PTransform on the given InputT, and returns its Output.
protected Coder<java.lang.Void>	getDefaultOutputCoder() Returns the default Coder to use for the output of this single-output PTransform.
void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
AvroIO.Write<T>	to(ResourceId outputPrefix) Writes to file(s) with the given output prefix.
AvroIO.Write<T>	to(java.lang.String outputPrefix) Writes to file(s) with the given output prefix.
AvroIO.Write<T>	to(ValueProvider<java.lang.String> outputPrefix) Like to(String).
AvroIO.Write<T>	toResource(ValueProvider<ResourceId> outputPrefix) Like to(ResourceId).
AvroIO.Write<T>	withCodec(CodecFactory codec) Writes to Avro file(s) compressed using specified codec.
AvroIO.Write<T>	withFilenamePolicy(FileBasedSink.FilenamePolicy filenamePolicy) Configures the FileBasedSink.FilenamePolicy that will be used to name written files.
AvroIO.Write<T>	withMetadata(java.util.Map<java.lang.String,java.lang.Object> metadata) Writes to Avro file(s) with the specified metadata.
AvroIO.Write<T>	withNumShards(int numShards) Configures the number of output shards produced overall (when using unwindowed writes) or per-window (when using windowed writes).
AvroIO.Write<T>	withoutSharding() Forces a single file as output and empty shard name template.
AvroIO.Write<T>	withShardNameTemplate(java.lang.String shardTemplate) Uses the given ShardNameTemplate for naming output files.
AvroIO.Write<T>	withSuffix(java.lang.String filenameSuffix) Configures the filename suffix for written files.
AvroIO.Write<T>	withWindowedWrites() Preserves windowing of input elements and writes them to files based on the element's window.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

to

public AvroIO.Write<T> to(java.lang.String outputPrefix)

Writes to file(s) with the given output prefix. See FileSystems for information on supported file systems.

The name of the output files will be determined by the FileBasedSink.FilenamePolicy used.

By default, a DefaultFilenamePolicy will build output filenames using the specified prefix, a shard name template (see withShardNameTemplate(String), and a common suffix (if supplied using withSuffix(String)). This default can be overridden using #withFilenamePolicy(FilenamePolicy).

to

@Experimental(value=FILESYSTEM)
public AvroIO.Write<T> to(ResourceId outputPrefix)

Writes to file(s) with the given output prefix. See FileSystems for information on supported file systems.

The name of the output files will be determined by the FileBasedSink.FilenamePolicy used.

By default, a DefaultFilenamePolicy will build output filenames using the specified prefix, a shard name template (see withShardNameTemplate(String), and a common suffix (if supplied using withSuffix(String)). This default can be overridden using #withFilenamePolicy(FilenamePolicy).

to

public AvroIO.Write<T> to(ValueProvider<java.lang.String> outputPrefix)

Like to(String).

toResource

@Experimental(value=FILESYSTEM)
public AvroIO.Write<T> toResource(ValueProvider<ResourceId> outputPrefix)

Like to(ResourceId).

withFilenamePolicy

public AvroIO.Write<T> withFilenamePolicy(FileBasedSink.FilenamePolicy filenamePolicy)

Configures the FileBasedSink.FilenamePolicy that will be used to name written files.

withShardNameTemplate

public AvroIO.Write<T> withShardNameTemplate(java.lang.String shardTemplate)

Uses the given ShardNameTemplate for naming output files. This option may only be used when #withFilenamePolicy(FilenamePolicy) has not been configured.

See DefaultFilenamePolicy for how the prefix, shard name template, and suffix are used.

withSuffix

public AvroIO.Write<T> withSuffix(java.lang.String filenameSuffix)

Configures the filename suffix for written files. This option may only be used when #withFilenamePolicy(FilenamePolicy) has not been configured.

See DefaultFilenamePolicy for how the prefix, shard name template, and suffix are used.

withNumShards

public AvroIO.Write<T> withNumShards(int numShards)

Configures the number of output shards produced overall (when using unwindowed writes) or per-window (when using windowed writes).

For unwindowed writes, constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

Parameters:

numShards - the number of shards to use, or 0 to let the system decide.

withoutSharding

public AvroIO.Write<T> withoutSharding()

Forces a single file as output and empty shard name template. This option is only compatible with unwindowed writes.

For unwindowed writes, constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

This is equivalent to .withNumShards(1).withShardNameTemplate("")

withWindowedWrites

public AvroIO.Write<T> withWindowedWrites()

Preserves windowing of input elements and writes them to files based on the element's window.

Requires use of withFilenamePolicy(FileBasedSink.FilenamePolicy). Filenames will be generated using FileBasedSink.FilenamePolicy.windowedFilename(org.apache.beam.sdk.io.fs.ResourceId, org.apache.beam.sdk.io.FileBasedSink.FilenamePolicy.WindowedContext, java.lang.String). See also WriteFiles.withWindowedWrites().

withCodec

public AvroIO.Write<T> withCodec(CodecFactory codec)

Writes to Avro file(s) compressed using specified codec.

withMetadata

public AvroIO.Write<T> withMetadata(java.util.Map<java.lang.String,java.lang.Object> metadata)

Writes to Avro file(s) with the specified metadata.

Supported value types are String, Long, and byte[].

expand

public PDone expand(PCollection<T> input)

Description copied from class: PTransform

Applies this PTransform on the given InputT, and returns its Output.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<T>,PDone>

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: PTransform

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class PTransform<PCollection<T>,PDone>

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData

getDefaultOutputCoder

protected Coder<java.lang.Void> getDefaultOutputCoder()

Description copied from class: PTransform

Returns the default Coder to use for the output of this single-output PTransform.

By default, always throws

Overrides:

getDefaultOutputCoder in class PTransform<PCollection<T>,PDone>