org.apache.beam.sdk.io

Class FileBasedSink.Writer<DestinationT,OutputT>

java.lang.Object

org.apache.beam.sdk.io.FileBasedSink.Writer<DestinationT,OutputT>

Type Parameters:

OutputT - the type of values to write.

Enclosing class:

FileBasedSink<UserT,DestinationT,OutputT>

public abstract static class FileBasedSink.Writer<DestinationT,OutputT>
extends java.lang.Object

Abstract writer that writes a bundle to a FileBasedSink. Subclass implementations provide a method that can write a single value to a WritableByteChannel.

Subclass implementations may also override methods that write headers and footers before and after the values in a bundle, respectively, as well as provide a MIME type for the output channel.

Multiple FileBasedSink.Writer instances may be created on the same worker, and therefore any access to static members or methods should be thread safe.

Constructor Summary

Constructors

Constructor and Description
Writer(FileBasedSink.WriteOperation<DestinationT,OutputT> writeOperation, java.lang.String mimeType) Construct a new FileBasedSink.Writer that will produce files of the given MIME type.

Method Summary

Modifier and Type	Method and Description
void	cleanup()
void	close() Closes the channel and returns the bundle result.
protected void	finishWrite() Called after all calls to writeHeader(), write(OutputT) and writeFooter().
DestinationT	getDestination() Return the user destination object for this writer.
ResourceId	getOutputFile()
FileBasedSink.WriteOperation<DestinationT,OutputT>	getWriteOperation() Return the WriteOperation that this Writer belongs to.
void	open(java.lang.String uId) Opens a uniquely named temporary file and initializes the writer using prepareWrite(java.nio.channels.WritableByteChannel).
protected abstract void	prepareWrite(java.nio.channels.WritableByteChannel channel) Called with the channel that a subclass will write its header, footer, and values to.
abstract void	write(OutputT value) Called for each value in the bundle.
protected void	writeFooter() Writes footer at the end of output files.
protected void	writeHeader() Writes header at the beginning of output files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Writer

public Writer(FileBasedSink.WriteOperation<DestinationT,OutputT> writeOperation,
              java.lang.String mimeType)

Construct a new FileBasedSink.Writer that will produce files of the given MIME type.

Method Detail

prepareWrite

protected abstract void prepareWrite(java.nio.channels.WritableByteChannel channel)
                              throws java.lang.Exception

Called with the channel that a subclass will write its header, footer, and values to. Subclasses should either keep a reference to the channel provided or create and keep a reference to an appropriate object that they will use to write to it.

Called before any subsequent calls to writeHeader, writeFooter, and write.

Throws:

java.lang.Exception

writeHeader

protected void writeHeader()
                    throws java.lang.Exception

Writes header at the beginning of output files. Nothing by default; subclasses may override.

Throws:

java.lang.Exception

writeFooter

protected void writeFooter()
                    throws java.lang.Exception

Writes footer at the end of output files. Nothing by default; subclasses may override.

Throws:

java.lang.Exception

finishWrite

protected void finishWrite()
                    throws java.lang.Exception

Called after all calls to writeHeader(), write(OutputT) and writeFooter(). If any resources opened in the write processes need to be flushed, flush them here.

Throws:

java.lang.Exception

open

public final void open(java.lang.String uId)
                throws java.lang.Exception

Opens a uniquely named temporary file and initializes the writer using prepareWrite(java.nio.channels.WritableByteChannel).

The unique id that is given to open should be used to ensure that the writer's output does not interfere with the output of other Writers, as a bundle may be executed many times for fault tolerance.

Throws:

java.lang.Exception

write

public abstract void write(OutputT value)
                    throws java.lang.Exception

Called for each value in the bundle.

Throws:

java.lang.Exception

getOutputFile

public ResourceId getOutputFile()

cleanup

public final void cleanup()
                   throws java.lang.Exception

Throws:

java.lang.Exception

close

public final void close()
                 throws java.lang.Exception

Closes the channel and returns the bundle result.

Throws:

java.lang.Exception

getWriteOperation

public FileBasedSink.WriteOperation<DestinationT,OutputT> getWriteOperation()

Return the WriteOperation that this Writer belongs to.

getDestination

public DestinationT getDestination()

Return the user destination object for this writer.