Package org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.bounded

Class FlinkBoundedSourceReader<T>

java.lang.Object

org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase<T,WindowedValue<T>>

org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.bounded.FlinkBoundedSourceReader<T>

Type Parameters:

T - the output element type of the encapsulated Beam BoundedReader.

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SourceReader<WindowedValue<T>,FlinkSourceSplit<T>>

public class FlinkBoundedSourceReader<T>
extends FlinkSourceReaderBase<T,WindowedValue<T>>

A Flink SourceReader implementation that reads from the assigned FlinkSourceSplits by using Beam BoundedReaders.

This reader consumes the source splits one by one sequentially, instead of concurrently.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase

FlinkSourceReaderBase.ReaderAndOutput

Field Summary

Fields inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase

AVAILABLE_NOW, context, idleTimeoutMs, invocationUtil, numRecordsInCounter, pipelineOptions, timestampExtractor

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	FlinkBoundedSourceReader(String stepName, org.apache.flink.api.connector.source.SourceReaderContext context, PipelineOptions pipelineOptions, ScheduledExecutorService executor, Function<WindowedValue<T>,Long> timestampExtractor)
	FlinkBoundedSourceReader(String stepName, org.apache.flink.api.connector.source.SourceReaderContext context, PipelineOptions pipelineOptions, Function<WindowedValue<T>,Long> timestampExtractor)

Method Summary

Modifier and Type	Method	Description
protected Source.Reader<T>	createReader(FlinkSourceSplit<T> sourceSplit)	Create Source.Reader for given FlinkSourceSplit.
protected FlinkSourceSplit<T>	getReaderCheckpoint(int splitId, FlinkSourceReaderBase<T,WindowedValue<T>>.ReaderAndOutput readerAndOutput)	Create FlinkSourceSplit for given splitId.
protected CompletableFuture<Void>	isAvailableForAliveReaders()	This method needs to be overridden by subclasses to determine if data is available when there are alive readers.
org.apache.flink.core.io.InputStatus	pollNext(org.apache.flink.api.connector.source.ReaderOutput<WindowedValue<T>> output)

Methods inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase

addSplits, addSplitsToUnfinishedForCheckpoint, allReaders, checkExceptionAndMaybeThrow, checkIdleTimeoutAndMaybeStartCountdown, close, createAndTrackNextReader, execute, finishSplit, hasException, ignoreReturnValue, isAvailable, noMoreSplits, notifyNoMoreSplits, recordException, scheduleTask, scheduleTaskAtFixedRate, snapshotState, sourceSplits, start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.api.connector.source.SourceReader

handleSourceEvents, notifyCheckpointComplete, pauseOrResumeSplits

Constructor Details

FlinkBoundedSourceReader

public FlinkBoundedSourceReader(String stepName,
 org.apache.flink.api.connector.source.SourceReaderContext context,
 PipelineOptions pipelineOptions,
 @Nullable
 Function<WindowedValue<T>,Long> timestampExtractor)

FlinkBoundedSourceReader

protected FlinkBoundedSourceReader(String stepName,
 org.apache.flink.api.connector.source.SourceReaderContext context,
 PipelineOptions pipelineOptions,
 ScheduledExecutorService executor,
 @Nullable
 Function<WindowedValue<T>,Long> timestampExtractor)

Method Details

getReaderCheckpoint

protected FlinkSourceSplit<T> getReaderCheckpoint(int splitId,
 FlinkSourceReaderBase<T,WindowedValue<T>>.ReaderAndOutput readerAndOutput)
                                           throws CoderException

Description copied from class: FlinkSourceReaderBase

Create FlinkSourceSplit for given splitId.

Specified by:

getReaderCheckpoint in class FlinkSourceReaderBase<T,WindowedValue<T>>

Throws:

CoderException

createReader

protected Source.Reader<T> createReader(@Nonnull
 FlinkSourceSplit<T> sourceSplit)
                                 throws IOException

Description copied from class: FlinkSourceReaderBase

Create Source.Reader for given FlinkSourceSplit.

Specified by:

createReader in class FlinkSourceReaderBase<T,WindowedValue<T>>

Throws:

IOException

pollNext

public org.apache.flink.core.io.InputStatus pollNext(org.apache.flink.api.connector.source.ReaderOutput<WindowedValue<T>> output)
                                              throws Exception

Throws:

Exception

isAvailableForAliveReaders

protected CompletableFuture<Void> isAvailableForAliveReaders()

Description copied from class: FlinkSourceReaderBase

This method needs to be overridden by subclasses to determine if data is available when there are alive readers. For example, an unbounded source may not have any source split ready for data emission even if all the sources are still alive. Whereas for the bounded source, data is always available as long as there are alive readers.

Specified by:

isAvailableForAliveReaders in class FlinkSourceReaderBase<T,WindowedValue<T>>