Package org.apache.beam.runners.spark.io

Class MicrobatchSource.Reader

java.lang.Object

org.apache.beam.sdk.io.Source.Reader<T>

org.apache.beam.runners.spark.io.MicrobatchSource.Reader

All Implemented Interfaces:

AutoCloseable

Enclosing class:

MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>

public class MicrobatchSource.Reader
extends Source.Reader<T>

Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

This Reader reads until one of the following thresholds has been reached:

1. max records (per batch)
2. max read duration (per batch)

Method Summary

Modifier and Type	Method	Description
boolean	advance()	Advances the reader to the next valid record.
void	close()	Closes the reader.
CheckpointMarkT	getCheckpointMark()
T	getCurrent()	Returns the value of the data item that was read by the last Source.Reader.start() or Source.Reader.advance() call.
Source<T>	getCurrentSource()	Returns a Source describing the same input that this Reader currently reads (including items already read).
Instant	getCurrentTimestamp()	Returns the timestamp associated with the current data item.
Instant	getWatermark()
boolean	start()	Initializes the reader and advances the reader to the first record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

start

public boolean start()
              throws IOException

Description copied from class: Source.Reader

Initializes the reader and advances the reader to the first record.

This method should be called exactly once. The invocation should occur prior to calling Source.Reader.advance() or Source.Reader.getCurrent(). This method may perform expensive operations that are needed to initialize the reader.

Specified by:

start in class Source.Reader<T>

Returns:

true if a record was read, false if there is no more input available.

Throws:

IOException

advance

public boolean advance()
                throws IOException

Description copied from class: Source.Reader

Advances the reader to the next valid record.

It is an error to call this without having called Source.Reader.start() first.

Specified by:

advance in class Source.Reader<T>

Returns:

true if a record was read, false if there is no more input available.

Throws:

IOException

getCurrent

public T getCurrent()
             throws NoSuchElementException

Description copied from class: Source.Reader

Returns the value of the data item that was read by the last Source.Reader.start() or Source.Reader.advance() call. The returned value must be effectively immutable and remain valid indefinitely.

Multiple calls to this method without an intervening call to Source.Reader.advance() should return the same result.

Specified by:

getCurrent in class Source.Reader<T>

Throws:

NoSuchElementException - if Source.Reader.start() was never called, or if the last Source.Reader.start() or Source.Reader.advance() returned false.

getCurrentTimestamp

public Instant getCurrentTimestamp()
                            throws NoSuchElementException

Description copied from class: Source.Reader

Returns the timestamp associated with the current data item.

If the source does not support timestamps, this should return BoundedWindow.TIMESTAMP_MIN_VALUE.

Multiple calls to this method without an intervening call to Source.Reader.advance() should return the same result.

Specified by:

getCurrentTimestamp in class Source.Reader<T>

Throws:

NoSuchElementException - if the reader is at the beginning of the input and Source.Reader.start() or Source.Reader.advance() wasn't called, or if the last Source.Reader.start() or Source.Reader.advance() returned false.

close

public void close()
           throws IOException

Description copied from class: Source.Reader

Closes the reader. The reader cannot be used after this method is called.

Specified by:

close in interface AutoCloseable

Specified by:

close in class Source.Reader<T>

Throws:

IOException

getCurrentSource

public Source<T> getCurrentSource()

Description copied from class: Source.Reader

Returns a Source describing the same input that this Reader currently reads (including items already read).

Usually, an implementation will simply return the immutable Source object from which the current Source.Reader was constructed, or delegate to the base class. However, when using or implementing this method on a BoundedSource.BoundedReader, special considerations apply, see documentation for BoundedSource.BoundedReader.getCurrentSource().

Specified by:

getCurrentSource in class Source.Reader<T>

getCheckpointMark

public CheckpointMarkT getCheckpointMark()

getWatermark

public Instant getWatermark()