Package org.apache.beam.runners.flink.translation.wrappers.streaming.io.source

Class LazyFlinkSourceSplitEnumerator<T>

java.lang.Object

org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.LazyFlinkSourceSplitEnumerator<T>

Type Parameters:

T - The output type of the encapsulated Beam Source.

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

public class LazyFlinkSourceSplitEnumerator<T>
extends Object
implements org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

A Flink SplitEnumerator implementation that holds a Beam Source and does the following:

• Split the Beam Source to desired number of splits.
• Lazily assign the splits to the Flink Source Reader.

Constructor Summary

Constructors

Constructor	Description
LazyFlinkSourceSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> context, Source<T> beamSource, PipelineOptions pipelineOptions, int numSplits, boolean splitInitialized)

Method Summary

Modifier and Type	Method	Description
void	addReader(int subtaskId)
void	addSplitsBack(List<FlinkSourceSplit<T>> splits, int subtaskId)
void	close()
void	handleSplitRequest(int subtask, String hostname)
void	initializeSplits()
Map<Integer,List<FlinkSourceSplit<T>>>	snapshotState()
Map<Integer,List<FlinkSourceSplit<T>>>	snapshotState(long checkpointId)
void	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.api.connector.source.SplitEnumerator

handleSourceEvent, notifyCheckpointComplete

Constructor Details

LazyFlinkSourceSplitEnumerator

public LazyFlinkSourceSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> context,
 Source<T> beamSource,
 PipelineOptions pipelineOptions,
 int numSplits,
 boolean splitInitialized)

Method Details

start

public void start()

Specified by:

start in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

initializeSplits

public void initializeSplits()

handleSplitRequest

public void handleSplitRequest(int subtask,
 @Nullable
 String hostname)

Specified by:

handleSplitRequest in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

addSplitsBack

public void addSplitsBack(List<FlinkSourceSplit<T>> splits,
 int subtaskId)

Specified by:

addSplitsBack in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

addReader

public void addReader(int subtaskId)

Specified by:

addReader in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

snapshotState

public Map<Integer,List<FlinkSourceSplit<T>>> snapshotState(long checkpointId)
                                                     throws Exception

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

Throws:

Exception

snapshotState

public Map<Integer,List<FlinkSourceSplit<T>>> snapshotState()
                                                     throws Exception

Throws:

Exception

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

Throws:

IOException