org.apache.beam.sdk.io

Class FileBasedSink.WriteOperation<DestinationT,OutputT>

java.lang.Object

org.apache.beam.sdk.io.FileBasedSink.WriteOperation<DestinationT,OutputT>

Type Parameters:

OutputT - the type of values written to the sink.

All Implemented Interfaces:

java.io.Serializable

Enclosing class:

FileBasedSink<UserT,DestinationT,OutputT>

public abstract static class FileBasedSink.WriteOperation<DestinationT,OutputT>
extends java.lang.Object
implements java.io.Serializable

Abstract operation that manages the process of writing to FileBasedSink.

The primary responsibilities of the WriteOperation is the management of output files. During a write, FileBasedSink.Writers write bundles to temporary file locations. After the bundles have been written,

1. finalize(java.lang.Iterable<org.apache.beam.sdk.io.FileBasedSink.FileResult<DestinationT>>) is given a list of the temporary files containing the output bundles.
2. During finalize, these temporary files are copied to final output locations and named according to a file naming template.
3. Finally, any temporary files that were created during the write are removed.

Subclass implementations of WriteOperation must implement createWriter() to return a concrete FileBasedSinkWriter.

Temporary and Output File Naming:

During the write, bundles are written to temporary files using the tempDirectory that can be provided via the constructor of WriteOperation. These temporary files will be named {tempDirectory}/{bundleId}, where bundleId is the unique id of the bundle. For example, if tempDirectory is "gs://my-bucket/my_temp_output", the output for a bundle with bundle id 15723 will be "gs://my-bucket/my_temp_output/15723".

Final output files are written to the location specified by the FileBasedSink.FilenamePolicy. If no filename policy is specified, then the DefaultFilenamePolicy will be used. The directory that the files are written to is determined by the FileBasedSink.FilenamePolicy instance.

Note that in the case of permanent failure of a bundle's write, no clean up of temporary files will occur.

If there are no elements in the PCollection being written, no output will be generated.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected FileBasedSink<?,DestinationT,OutputT>	sink The Sink that this WriteOperation will write to.
protected ValueProvider<ResourceId>	tempDirectory Directory for temporary output files.
protected boolean	windowedWrites Whether windowed writes are being used.

Constructor Summary

Constructors

Constructor and Description
WriteOperation(FileBasedSink<?,DestinationT,OutputT> sink) Constructs a WriteOperation using the default strategy for generating a temporary directory from the base output filename.
WriteOperation(FileBasedSink<?,DestinationT,OutputT> sink, ResourceId tempDirectory) Create a new WriteOperation.

Method Summary

Modifier and Type	Method and Description
protected java.util.Map<ResourceId,ResourceId>	buildOutputFilenames(java.lang.Iterable<FileBasedSink.FileResult<DestinationT>> writerResults)
protected static ResourceId	buildTemporaryFilename(ResourceId tempDirectory, java.lang.String filename) Constructs a temporary file resource given the temporary directory and a filename.
abstract FileBasedSink.Writer<DestinationT,OutputT>	createWriter() Clients must implement to return a subclass of FileBasedSink.Writer.
java.util.Map<ResourceId,ResourceId>	finalize(java.lang.Iterable<FileBasedSink.FileResult<DestinationT>> writerResults) Finalizes writing by copying temporary output files to their final location.
FileBasedSink<?,DestinationT,OutputT>	getSink() Returns the FileBasedSink for this write operation.
void	removeTemporaryFiles(java.util.Set<ResourceId> filenames)
void	setWindowedWrites(boolean windowedWrites) Indicates that the operation will be performing windowed writes.
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

sink

protected final FileBasedSink<?,DestinationT,OutputT> sink

The Sink that this WriteOperation will write to.

tempDirectory

protected final ValueProvider<ResourceId> tempDirectory

Directory for temporary output files.

windowedWrites

@Experimental(value=FILESYSTEM)
protected boolean windowedWrites

Whether windowed writes are being used.

Constructor Detail

WriteOperation

public WriteOperation(FileBasedSink<?,DestinationT,OutputT> sink)

Constructs a WriteOperation using the default strategy for generating a temporary directory from the base output filename.

Default is a uniquely named subdirectory of the provided tempDirectory, e.g. if tempDirectory is /path/to/foo/, the temporary directory will be /path/to/foo/temp-beam-foo-$date.

Parameters:

sink - the FileBasedSink that will be used to configure this write operation.

WriteOperation

@Experimental(value=FILESYSTEM)
public WriteOperation(FileBasedSink<?,DestinationT,OutputT> sink,
                                                      ResourceId tempDirectory)

Create a new WriteOperation.

Parameters:

sink - the FileBasedSink that will be used to configure this write operation.

tempDirectory - the base directory to be used for temporary output files.

Method Detail

buildTemporaryFilename

@Experimental(value=FILESYSTEM)
protected static ResourceId buildTemporaryFilename(ResourceId tempDirectory,
                                                                                   java.lang.String filename)
                                                                            throws java.io.IOException

Constructs a temporary file resource given the temporary directory and a filename.

Throws:

java.io.IOException

createWriter

public abstract FileBasedSink.Writer<DestinationT,OutputT> createWriter()
                                                                 throws java.lang.Exception

Clients must implement to return a subclass of FileBasedSink.Writer. This method must not mutate the state of the object.

Throws:

java.lang.Exception

setWindowedWrites

public void setWindowedWrites(boolean windowedWrites)

Indicates that the operation will be performing windowed writes.

finalize

public java.util.Map<ResourceId,ResourceId> finalize(java.lang.Iterable<FileBasedSink.FileResult<DestinationT>> writerResults)
                                              throws java.lang.Exception

Finalizes writing by copying temporary output files to their final location.

Finalization may be overridden by subclass implementations to perform customized finalization (e.g., initiating some operation on output bundles, merging them, etc.). writerResults contains the filenames of written bundles.

If subclasses override this method, they must guarantee that its implementation is idempotent, as it may be executed multiple times in the case of failure or for redundancy. It is a best practice to attempt to try to make this method atomic.

Returns the map of temporary files generated to final filenames. Callers must call removeTemporaryFiles(Set) to cleanup the temporary files.

Parameters:

writerResults - the results of writes (FileResult).

Throws:

java.lang.Exception

removeTemporaryFiles

public void removeTemporaryFiles(java.util.Set<ResourceId> filenames)
                          throws java.io.IOException

Throws:

java.io.IOException

buildOutputFilenames

@Experimental(value=FILESYSTEM)
protected final java.util.Map<ResourceId,ResourceId> buildOutputFilenames(java.lang.Iterable<FileBasedSink.FileResult<DestinationT>> writerResults)

getSink

public FileBasedSink<?,DestinationT,OutputT> getSink()

Returns the FileBasedSink for this write operation.

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object