Package org.apache.beam.runners.flink.translation.wrappers.streaming.io.source

Class FlinkSource<T,OutputT>

java.lang.Object

org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSource<T,OutputT>

Type Parameters:

T - The data type of the records emitted by the raw Beam sources.

OutputT - The data type of the records emitted by the Flink Source.

All Implemented Interfaces:

Serializable, org.apache.flink.api.connector.source.Source<OutputT,FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>, org.apache.flink.api.connector.source.SourceReaderFactory<OutputT,FlinkSourceSplit<T>>

Direct Known Subclasses:

FlinkBoundedSource, FlinkUnboundedSource

public abstract class FlinkSource<T,OutputT>
extends Object
implements org.apache.flink.api.connector.source.Source<OutputT,FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

The base class for FlinkBoundedSource and FlinkUnboundedSource.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	FlinkSource.TimestampExtractor<T>

Field Summary

Fields

Modifier and Type	Field	Description
protected final Source<T>	beamSource
protected final org.apache.flink.api.connector.source.Boundedness	boundedness
protected final org.apache.beam.runners.core.construction.SerializablePipelineOptions	serializablePipelineOptions
protected final String	stepName

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	FlinkSource(String stepName, Source<T> beamSource, org.apache.beam.runners.core.construction.SerializablePipelineOptions serializablePipelineOptions, org.apache.flink.api.connector.source.Boundedness boundedness, int numSplits)

Method Summary

Modifier and Type	Method	Description
static <T> FlinkBoundedSource<T>	bounded(String stepName, BoundedSource<T> boundedSource, org.apache.beam.runners.core.construction.SerializablePipelineOptions serializablePipelineOptions, int numSplits)
static FlinkBoundedSource<byte[]>	boundedImpulse()
org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>	createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> enumContext)
org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>	createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> enumContext, boolean splitInitialized)
org.apache.flink.api.connector.source.Boundedness	getBoundedness()
org.apache.flink.core.io.SimpleVersionedSerializer<Map<Integer,List<FlinkSourceSplit<T>>>>	getEnumeratorCheckpointSerializer()
int	getNumSplits()
org.apache.flink.core.io.SimpleVersionedSerializer<FlinkSourceSplit<T>>	getSplitSerializer()
org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>	restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> enumContext, Map<Integer,List<FlinkSourceSplit<T>>> checkpoint)
static <T> FlinkUnboundedSource<T>	unbounded(String stepName, UnboundedSource<T,?> source, org.apache.beam.runners.core.construction.SerializablePipelineOptions serializablePipelineOptions, int numSplits)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.source.SourceReaderFactory

createReader

Field Details

stepName

protected final String stepName

beamSource

protected final Source<T> beamSource

boundedness

protected final org.apache.flink.api.connector.source.Boundedness boundedness

serializablePipelineOptions

protected final org.apache.beam.runners.core.construction.SerializablePipelineOptions serializablePipelineOptions

Constructor Details

FlinkSource

protected FlinkSource(String stepName,
 Source<T> beamSource,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions serializablePipelineOptions,
 org.apache.flink.api.connector.source.Boundedness boundedness,
 int numSplits)

Method Details

bounded

public static <T> FlinkBoundedSource<T> bounded(String stepName,
 BoundedSource<T> boundedSource,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions serializablePipelineOptions,
 int numSplits)

unbounded

public static <T> FlinkUnboundedSource<T> unbounded(String stepName,
 UnboundedSource<T,?> source,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions serializablePipelineOptions,
 int numSplits)

boundedImpulse

public static FlinkBoundedSource<byte[]> boundedImpulse()

getBoundedness

public org.apache.flink.api.connector.source.Boundedness getBoundedness()

Specified by:

getBoundedness in interface org.apache.flink.api.connector.source.Source<OutputT,FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

createEnumerator

public org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>> createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> enumContext)
                                                                                                                                   throws Exception

Specified by:

createEnumerator in interface org.apache.flink.api.connector.source.Source<OutputT,FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

Throws:

Exception

createEnumerator

public org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>> createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> enumContext,
 boolean splitInitialized)
                                                                                                                                   throws Exception

Throws:

Exception

restoreEnumerator

public org.apache.flink.api.connector.source.SplitEnumerator<FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>> restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FlinkSourceSplit<T>> enumContext,
 Map<Integer,List<FlinkSourceSplit<T>>> checkpoint)
                                                                                                                                    throws Exception

Specified by:

restoreEnumerator in interface org.apache.flink.api.connector.source.Source<OutputT,FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

Throws:

Exception

getSplitSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<FlinkSourceSplit<T>> getSplitSerializer()

Specified by:

getSplitSerializer in interface org.apache.flink.api.connector.source.Source<OutputT,FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

getEnumeratorCheckpointSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<Map<Integer,List<FlinkSourceSplit<T>>>> getEnumeratorCheckpointSerializer()

Specified by:

getEnumeratorCheckpointSerializer in interface org.apache.flink.api.connector.source.Source<OutputT,FlinkSourceSplit<T>,Map<Integer,List<FlinkSourceSplit<T>>>>

getNumSplits

public int getNumSplits()