Package org.apache.beam.runners.flink.translation.wrappers.streaming.io.source

Class FlinkSourceSplitEnumerator<T>

java.lang.Object

org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceSplitEnumerator<T>

Type Parameters:

T - The output type of the encapsulated Beam Source.

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

public class FlinkSourceSplitEnumerator<T>
extends Object
implements org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

A Flink SplitEnumerator implementation that holds a Beam Source and does the following:

• Split the Beam Source to desired number of splits.
• Assign the splits to the Flink Source Reader.

Note that at this point, this class has a static round-robin split assignment strategy.

Constructor Summary

Constructors

Constructor	Description
FlinkSourceSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> context, Source<T> beamSource, PipelineOptions pipelineOptions, int numSplits)
FlinkSourceSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> context, Source<T> beamSource, PipelineOptions pipelineOptions, int numSplits, boolean splitsInitialized)

Method Summary

Modifier and Type	Method	Description
void	addReader(int subtaskId)
void	addSplitsBack(List<FlinkSourceSplit<T>> splits, int subtaskId)
void	close()
void	handleSplitRequest(int subtaskId, String requesterHostname)
Map<Integer,List<FlinkSourceSplit<T>>>	snapshotState(long checkpointId)
void	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.api.connector.source.SplitEnumerator

handleSourceEvent, notifyCheckpointComplete

Constructor Details

FlinkSourceSplitEnumerator

public FlinkSourceSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> context,
 Source<T> beamSource,
 PipelineOptions pipelineOptions,
 int numSplits)

FlinkSourceSplitEnumerator

public FlinkSourceSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> context,
 Source<T> beamSource,
 PipelineOptions pipelineOptions,
 int numSplits,
 boolean splitsInitialized)

Method Details

start

public void start()

Specified by:

start in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

handleSplitRequest

public void handleSplitRequest(int subtaskId,
 @Nullable
 String requesterHostname)

Specified by:

handleSplitRequest in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

addSplitsBack

public void addSplitsBack(List<FlinkSourceSplit<T>> splits,
 int subtaskId)

Specified by:

addSplitsBack in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

addReader

public void addReader(int subtaskId)

Specified by:

addReader in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

snapshotState

public Map<Integer,List<FlinkSourceSplit<T>>> snapshotState(long checkpointId)
                                                     throws Exception

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

Throws:

Exception

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

Throws:

IOException