Package org.apache.beam.sdk.extensions.avro.io

Class AvroIO.Write<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<T>,PDone>

org.apache.beam.sdk.extensions.avro.io.AvroIO.Write<T>

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

AvroIO

public static class AvroIO.Write<T>
extends PTransform<PCollection<T>,PDone>

This class is used as the default return value of AvroIO.write(java.lang.Class<T>)

All methods in this class delegate to the appropriate method of AvroIO.TypedWrite. This class exists for backwards compatibility, and will be removed in Beam 3.0.

See Also:

• Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Method Summary

Modifier and Type	Method	Description
PDone	expand(PCollection<T> input)	Override this method to specify how this PTransform should be expanded on the given InputT.
void	populateDisplayData(DisplayData.Builder builder)	Register display data for the given transform or component.
AvroIO.Write<T>	to(String outputPrefix)	See AvroIO.TypedWrite.to(String).
AvroIO.Write<T>	to(DynamicAvroDestinations<T,?,T> dynamicDestinations)	Deprecated. Use FileIO.write() or FileIO.writeDynamic() instead.
AvroIO.Write<T>	to(FileBasedSink.FilenamePolicy filenamePolicy)	See AvroIO.TypedWrite.to(FilenamePolicy).
AvroIO.Write<T>	to(ResourceId outputPrefix)	See AvroIO.TypedWrite.to(ResourceId) .
AvroIO.Write<T>	to(ValueProvider<String> outputPrefix)	See AvroIO.TypedWrite.to(ValueProvider).
AvroIO.Write<T>	toResource(ValueProvider<ResourceId> outputPrefix)	See AvroIO.TypedWrite.to(ResourceId).
AvroIO.Write<T>	withCodec(CodecFactory codec)	See AvroIO.TypedWrite.withCodec(org.apache.avro.file.CodecFactory).
AvroIO.Write<T>	withDatumWriterFactory(AvroSink.DatumWriterFactory<T> datumWriterFactory)	See AvroIO.TypedWrite.withDatumWriterFactory(org.apache.beam.sdk.extensions.avro.io.AvroSink.DatumWriterFactory<OutputT>).
AvroIO.Write<T>	withMetadata(Map<String,Object> metadata)	See AvroIO.TypedWrite.withMetadata(java.util.Map<java.lang.String, java.lang.Object>) .
AvroIO.Write<T>	withNumShards(int numShards)	See AvroIO.TypedWrite.withNumShards(int).
<DestinationT> AvroIO.TypedWrite<T,DestinationT,T>	withOutputFilenames()	Specify that output filenames are wanted.
AvroIO.Write<T>	withoutSharding()	See AvroIO.TypedWrite.withoutSharding().
AvroIO.Write<T>	withSchema(Schema schema)	See AvroIO.TypedWrite.withSchema(org.apache.avro.Schema).
AvroIO.Write<T>	withShardNameTemplate(String shardTemplate)	See AvroIO.TypedWrite.withShardNameTemplate(java.lang.String).
AvroIO.Write<T>	withSuffix(String filenameSuffix)	See AvroIO.TypedWrite.withSuffix(java.lang.String).
AvroIO.Write<T>	withSyncInterval(int syncInterval)	See AvroIO.TypedWrite.withSyncInterval(int).
AvroIO.Write<T>	withTempDirectory(ResourceId tempDirectory)	See AvroIO.TypedWrite.withTempDirectory(ResourceId).
AvroIO.Write<T>	withTempDirectory(ValueProvider<ResourceId> tempDirectory)	See AvroIO.TypedWrite.withTempDirectory(ValueProvider).
AvroIO.Write<T>	withWindowedWrites()	See AvroIO.TypedWrite.withWindowedWrites().

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Details

to

public AvroIO.Write<T> to(String outputPrefix)

See AvroIO.TypedWrite.to(String).

to

public AvroIO.Write<T> to(ResourceId outputPrefix)

See AvroIO.TypedWrite.to(ResourceId) .

to

public AvroIO.Write<T> to(ValueProvider<String> outputPrefix)

See AvroIO.TypedWrite.to(ValueProvider).

toResource

public AvroIO.Write<T> toResource(ValueProvider<ResourceId> outputPrefix)

See AvroIO.TypedWrite.to(ResourceId).

to

public AvroIO.Write<T> to(FileBasedSink.FilenamePolicy filenamePolicy)

See AvroIO.TypedWrite.to(FilenamePolicy).

to

@Deprecated
public AvroIO.Write<T> to(DynamicAvroDestinations<T,?,T> dynamicDestinations)

Deprecated.

Use FileIO.write() or FileIO.writeDynamic() instead.

See AvroIO.TypedWrite.to(DynamicAvroDestinations).

withSyncInterval

public AvroIO.Write<T> withSyncInterval(int syncInterval)

See AvroIO.TypedWrite.withSyncInterval(int).

withSchema

public AvroIO.Write<T> withSchema(Schema schema)

See AvroIO.TypedWrite.withSchema(org.apache.avro.Schema).

withTempDirectory

public AvroIO.Write<T> withTempDirectory(ValueProvider<ResourceId> tempDirectory)

See AvroIO.TypedWrite.withTempDirectory(ValueProvider).

withTempDirectory

public AvroIO.Write<T> withTempDirectory(ResourceId tempDirectory)

See AvroIO.TypedWrite.withTempDirectory(ResourceId).

withShardNameTemplate

public AvroIO.Write<T> withShardNameTemplate(String shardTemplate)

See AvroIO.TypedWrite.withShardNameTemplate(java.lang.String).

withSuffix

public AvroIO.Write<T> withSuffix(String filenameSuffix)

See AvroIO.TypedWrite.withSuffix(java.lang.String).

withNumShards

public AvroIO.Write<T> withNumShards(int numShards)

See AvroIO.TypedWrite.withNumShards(int).

withoutSharding

public AvroIO.Write<T> withoutSharding()

See AvroIO.TypedWrite.withoutSharding().

withWindowedWrites

public AvroIO.Write<T> withWindowedWrites()

See AvroIO.TypedWrite.withWindowedWrites().

withCodec

public AvroIO.Write<T> withCodec(CodecFactory codec)

See AvroIO.TypedWrite.withCodec(org.apache.avro.file.CodecFactory).

withDatumWriterFactory

public AvroIO.Write<T> withDatumWriterFactory(AvroSink.DatumWriterFactory<T> datumWriterFactory)

See AvroIO.TypedWrite.withDatumWriterFactory(org.apache.beam.sdk.extensions.avro.io.AvroSink.DatumWriterFactory<OutputT>).

withOutputFilenames

public <DestinationT>
AvroIO.TypedWrite<T,DestinationT,T> withOutputFilenames()

Specify that output filenames are wanted.

The nested AvroIO.TypedWritetransform always has access to output filenames, however due to backwards-compatibility concerns, AvroIO.Write cannot return them. This method simply returns the inner AvroIO.TypedWrite transform which has WriteFilesResult as its output type, allowing access to output files.

The supplied DestinationT type must be: the same as that supplied in to(DynamicAvroDestinations) if that method was used, or Void otherwise.

withMetadata

public AvroIO.Write<T> withMetadata(Map<String,Object> metadata)

See AvroIO.TypedWrite.withMetadata(java.util.Map<java.lang.String, java.lang.Object>) .

expand

public PDone expand(PCollection<T> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<T>,PDone>

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: PTransform

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class PTransform<PCollection<T>,PDone>

Parameters:

builder - The builder to populate with display data.

See Also:

• HasDisplayData