Package org.apache.beam.sdk.io

Class FileBasedSink.Writer<DestinationT,OutputT>

java.lang.Object
org.apache.beam.sdk.io.FileBasedSink.Writer<DestinationT,OutputT>
Type Parameters:
OutputT - the type of values to write.
Enclosing class:
FileBasedSink<UserT,DestinationT,OutputT>
public abstract static class FileBasedSink.Writer<DestinationT,OutputT> extends Object
Abstract writer that writes a bundle to a FileBasedSink. Subclass implementations provide a method that can write a single value to a WritableByteChannel.

Subclass implementations may also override methods that write headers and footers before and after the values in a bundle, respectively, as well as provide a MIME type for the output channel.

Multiple FileBasedSink.Writer instances may be created on the same worker, and therefore any access to static members or methods should be thread safe.

  • Constructor Details

  • Method Details

    • prepareWrite

      protected abstract void prepareWrite(WritableByteChannel channel) throws Exception
      Called with the channel that a subclass will write its header, footer, and values to. Subclasses should either keep a reference to the channel provided or create and keep a reference to an appropriate object that they will use to write to it.

      Called before any subsequent calls to writeHeader, writeFooter, and write.

      Throws:
      Exception

    • writeHeader

      protected void writeHeader() throws Exception
      Writes header at the beginning of output files. Nothing by default; subclasses may override.
      Throws:
      Exception

    • writeFooter

      protected void writeFooter() throws Exception
      Writes footer at the end of output files. Nothing by default; subclasses may override.
      Throws:
      Exception

    • finishWrite

      protected void finishWrite() throws Exception
      Called after all calls to writeHeader(), write(OutputT) and writeFooter(). If any resources opened in the write processes need to be flushed, flush them here.
      Throws:
      Exception

    • open

      public final void open(String uId) throws Exception
      Opens a uniquely named temporary file and initializes the writer using prepareWrite(java.nio.channels.WritableByteChannel).

      The unique id that is given to open should be used to ensure that the writer's output does not interfere with the output of other Writers, as a bundle may be executed many times for fault tolerance.

      Throws:
      Exception

    • write

      public abstract void write(OutputT value) throws Exception
      Called for each value in the bundle.
      Throws:
      Exception

    • getOutputFile

      public ResourceId getOutputFile()

    • cleanup

      public final void cleanup() throws Exception
      Throws:
      Exception

    • close

      public final void close() throws Exception
      Closes the channel and returns the bundle result.
      Throws:
      Exception

    • getWriteOperation

      public FileBasedSink.WriteOperation<DestinationT,OutputT> getWriteOperation()
      Return the WriteOperation that this Writer belongs to.

    • getDestination

      public DestinationT getDestination()
      Return the user destination object for this writer.