org.apache.beam.sdk.io

Class FileBasedSink.Writer<DestinationT,OutputT>

java.lang.Object

org.apache.beam.sdk.io.FileBasedSink.Writer<DestinationT,OutputT>

Type Parameters:

OutputT - the type of values to write.

Enclosing class:

FileBasedSink<UserT,DestinationT,OutputT>

public abstract static class FileBasedSink.Writer<DestinationT,OutputT>
extends java.lang.Object

Abstract writer that writes a bundle to a FileBasedSink. Subclass implementations provide a method that can write a single value to a WritableByteChannel.

Subclass implementations may also override methods that write headers and footers before and after the values in a bundle, respectively, as well as provide a MIME type for the output channel.

Multiple FileBasedSink.Writer instances may be created on the same worker, and therefore any access to static members or methods should be thread safe.

Constructor Summary

Constructors

Constructor and Description
Writer(FileBasedSink.WriteOperation<DestinationT,OutputT> writeOperation, java.lang.String mimeType) Construct a new FileBasedSink.Writer that will produce files of the given MIME type.

Method Summary

Modifier and Type	Method and Description
void	cleanup()
FileBasedSink.FileResult<DestinationT>	close() Closes the channel and returns the bundle result.
protected void	finishWrite() Called after all calls to writeHeader(), write(OutputT) and writeFooter().
DestinationT	getDestination() Return the user destination object for this writer.
FileBasedSink.WriteOperation<DestinationT,OutputT>	getWriteOperation() Return the WriteOperation that this Writer belongs to.
void	openUnwindowed(java.lang.String uId, int shard, DestinationT destination) Similar to openWindowed(java.lang.String, org.apache.beam.sdk.transforms.windowing.BoundedWindow, org.apache.beam.sdk.transforms.windowing.PaneInfo, int, DestinationT) however for the case where unwindowed writes were requested.
void	openWindowed(java.lang.String uId, BoundedWindow window, PaneInfo paneInfo, int shard, DestinationT destination) Performs bundle initialization.
protected abstract void	prepareWrite(java.nio.channels.WritableByteChannel channel) Called with the channel that a subclass will write its header, footer, and values to.
abstract void	write(OutputT value) Called for each value in the bundle.
protected void	writeFooter() Writes footer at the end of output files.
protected void	writeHeader() Writes header at the beginning of output files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Writer

public Writer(FileBasedSink.WriteOperation<DestinationT,OutputT> writeOperation,
              java.lang.String mimeType)

Construct a new FileBasedSink.Writer that will produce files of the given MIME type.

Method Detail

prepareWrite

protected abstract void prepareWrite(java.nio.channels.WritableByteChannel channel)
                              throws java.lang.Exception

Called with the channel that a subclass will write its header, footer, and values to. Subclasses should either keep a reference to the channel provided or create and keep a reference to an appropriate object that they will use to write to it.

Called before any subsequent calls to writeHeader, writeFooter, and write.

Throws:

java.lang.Exception

writeHeader

protected void writeHeader()
                    throws java.lang.Exception

Writes header at the beginning of output files. Nothing by default; subclasses may override.

Throws:

java.lang.Exception

writeFooter

protected void writeFooter()
                    throws java.lang.Exception

Writes footer at the end of output files. Nothing by default; subclasses may override.

Throws:

java.lang.Exception

finishWrite

protected void finishWrite()
                    throws java.lang.Exception

Called after all calls to writeHeader(), write(OutputT) and writeFooter(). If any resources opened in the write processes need to be flushed, flush them here.

Throws:

java.lang.Exception

openWindowed

public final void openWindowed(java.lang.String uId,
                               BoundedWindow window,
                               PaneInfo paneInfo,
                               int shard,
                               DestinationT destination)
                        throws java.lang.Exception

Performs bundle initialization. For example, creates a temporary file for writing or initializes any state that will be used across calls to write(OutputT).

The unique id that is given to open should be used to ensure that the writer's output does not interfere with the output of other Writers, as a bundle may be executed many times for fault tolerance.

The window and paneInfo arguments are populated when windowed writes are requested. shard id populated for the case of static sharding. In cases where the runner is dynamically picking sharding, shard might be set to -1.

Throws:

java.lang.Exception

write

public abstract void write(OutputT value)
                    throws java.lang.Exception

Called for each value in the bundle.

Throws:

java.lang.Exception

openUnwindowed

public final void openUnwindowed(java.lang.String uId,
                                 int shard,
                                 DestinationT destination)
                          throws java.lang.Exception

Similar to openWindowed(java.lang.String, org.apache.beam.sdk.transforms.windowing.BoundedWindow, org.apache.beam.sdk.transforms.windowing.PaneInfo, int, DestinationT) however for the case where unwindowed writes were requested.

Throws:

java.lang.Exception

cleanup

public final void cleanup()
                   throws java.lang.Exception

Throws:

java.lang.Exception

close

public final FileBasedSink.FileResult<DestinationT> close()
                                                   throws java.lang.Exception

Closes the channel and returns the bundle result.

Throws:

java.lang.Exception

getWriteOperation

public FileBasedSink.WriteOperation<DestinationT,OutputT> getWriteOperation()

Return the WriteOperation that this Writer belongs to.

getDestination

public DestinationT getDestination()

Return the user destination object for this writer.