org.apache.beam.sdk.io

Class FileBasedSink<UserT,DestinationT,OutputT>

java.lang.Object

org.apache.beam.sdk.io.FileBasedSink<UserT,DestinationT,OutputT>

Type Parameters:

OutputT - the type of values written to the sink.

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Direct Known Subclasses:

AvroSink

public abstract class FileBasedSink<UserT,DestinationT,OutputT>
extends java.lang.Object
implements java.io.Serializable, HasDisplayData

Abstract class for file-based output. An implementation of FileBasedSink writes file-based output and defines the format of output files (how values are written, headers/footers, MIME type, etc.).

At pipeline construction time, the methods of FileBasedSink are called to validate the sink and to create a FileBasedSink.WriteOperation that manages the process of writing to the sink.

The process of writing to file-based sink is as follows:

1. An optional subclass-defined initialization,
2. a parallel write of bundles to temporary files, and finally,
3. these temporary files are renamed with final output filenames.

In order to ensure fault-tolerance, a bundle may be executed multiple times (e.g., in the event of failure/retry or for redundancy). However, exactly one of these executions will have its result passed to the finalize method. Each call to FileBasedSink.Writer.open(java.lang.String) is passed a unique bundle id when it is called by the WriteFiles transform, so even redundant or retried bundles will have a unique way of identifying their output.

The bundle id should be used to guarantee that a bundle's output is unique. This uniqueness guarantee is important; if a bundle is to be output to a file, for example, the name of the file will encode the unique bundle id to avoid conflicts with other writers.

FileBasedSink can take a custom FileBasedSink.FilenamePolicy object to determine output filenames, and this policy object can be used to write windowed or triggered PCollections into separate files per window pane. This allows file output from unbounded PCollections, and also works for bounded PCollections.

Supported file systems are those registered with FileSystems.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileBasedSink.CompressionType Deprecated. use Compression.
static class	FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT> A class that allows value-dependent writes in FileBasedSink.
static class	FileBasedSink.FilenamePolicy A naming policy for output files.
static class	FileBasedSink.FileResult<DestinationT> Result of a single bundle write.
static class	FileBasedSink.FileResultCoder<DestinationT> A coder for FileBasedSink.FileResult objects.
static interface	FileBasedSink.OutputFileHints Provides hints about how to generate output files, such as a suggested filename suffix (e.g.
static interface	FileBasedSink.WritableByteChannelFactory Implementations create instances of WritableByteChannel used by FileBasedSink and related classes to allow decorating, or otherwise transforming, the raw data that would normally be written directly to the WritableByteChannel passed into FileBasedSink.WritableByteChannelFactory.create(WritableByteChannel).
static class	FileBasedSink.WriteOperation<DestinationT,OutputT> Abstract operation that manages the process of writing to FileBasedSink.
static class	FileBasedSink.Writer<DestinationT,OutputT> Abstract writer that writes a bundle to a FileBasedSink.

Constructor Summary

Constructors

Constructor and Description
FileBasedSink(ValueProvider<ResourceId> tempDirectoryProvider, FileBasedSink.DynamicDestinations<?,DestinationT,OutputT> dynamicDestinations) Construct a FileBasedSink with the given temp directory, producing uncompressed files.
FileBasedSink(ValueProvider<ResourceId> tempDirectoryProvider, FileBasedSink.DynamicDestinations<?,DestinationT,OutputT> dynamicDestinations, Compression compression) Construct a FileBasedSink with the given temp directory and output channel type.
FileBasedSink(ValueProvider<ResourceId> tempDirectoryProvider, FileBasedSink.DynamicDestinations<?,DestinationT,OutputT> dynamicDestinations, FileBasedSink.WritableByteChannelFactory writableByteChannelFactory) Construct a FileBasedSink with the given temp directory and output channel type.

Method Summary

Modifier and Type	Method and Description
static ResourceId	convertToFileResourceIfPossible(java.lang.String outputPrefix) This is a helper function for turning a user-provided output filename prefix and converting it into a ResourceId for writing output files.
abstract FileBasedSink.WriteOperation<DestinationT,OutputT>	createWriteOperation() Return a subclass of FileBasedSink.WriteOperation that will manage the write to the sink.
FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT>	getDynamicDestinations() Return the FileBasedSink.DynamicDestinations used.
ValueProvider<ResourceId>	getTempDirectoryProvider() Returns the directory inside which temporary files will be written according to the configured FileBasedSink.FilenamePolicy.
protected FileBasedSink.WritableByteChannelFactory	getWritableByteChannelFactory() Returns the FileBasedSink.WritableByteChannelFactory used.
void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
void	validate(PipelineOptions options)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileBasedSink

public FileBasedSink(ValueProvider<ResourceId> tempDirectoryProvider,
                     FileBasedSink.DynamicDestinations<?,DestinationT,OutputT> dynamicDestinations)

Construct a FileBasedSink with the given temp directory, producing uncompressed files.

FileBasedSink

public FileBasedSink(ValueProvider<ResourceId> tempDirectoryProvider,
                     FileBasedSink.DynamicDestinations<?,DestinationT,OutputT> dynamicDestinations,
                     FileBasedSink.WritableByteChannelFactory writableByteChannelFactory)

Construct a FileBasedSink with the given temp directory and output channel type.

FileBasedSink

public FileBasedSink(ValueProvider<ResourceId> tempDirectoryProvider,
                     FileBasedSink.DynamicDestinations<?,DestinationT,OutputT> dynamicDestinations,
                     Compression compression)

Construct a FileBasedSink with the given temp directory and output channel type.

Method Detail

convertToFileResourceIfPossible

public static ResourceId convertToFileResourceIfPossible(java.lang.String outputPrefix)

This is a helper function for turning a user-provided output filename prefix and converting it into a ResourceId for writing output files. See TextIO.Write.to(String) for an example use case.

Typically, the input prefix will be something like /tmp/foo/bar, and the user would like output files to be named as /tmp/foo/bar-0-of-3.txt. Thus, this function tries to interpret the provided string as a file ResourceId path.

However, this may fail, for example if the user gives a prefix that is a directory. E.g., /, gs://my-bucket, or c://. In that case, interpreting the string as a file will fail and this function will return a directory ResourceId instead.

getDynamicDestinations

public FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT> getDynamicDestinations()

Return the FileBasedSink.DynamicDestinations used.

getTempDirectoryProvider

public ValueProvider<ResourceId> getTempDirectoryProvider()

Returns the directory inside which temporary files will be written according to the configured FileBasedSink.FilenamePolicy.

validate

public void validate(PipelineOptions options)

createWriteOperation

public abstract FileBasedSink.WriteOperation<DestinationT,OutputT> createWriteOperation()

Return a subclass of FileBasedSink.WriteOperation that will manage the write to the sink.

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from interface: HasDisplayData

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

Specified by:

populateDisplayData in interface HasDisplayData

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData

getWritableByteChannelFactory

protected final FileBasedSink.WritableByteChannelFactory getWritableByteChannelFactory()

Returns the FileBasedSink.WritableByteChannelFactory used.