Package org.apache.beam.runners.flink.translation.wrappers.streaming.io

Class UnboundedSourceWrapper<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<WindowedValue<ValueWithRecordId<OutputT>>>

org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper<OutputT,CheckpointMarkT>

All Implemented Interfaces:

Serializable, BeamStoppableFunction, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<WindowedValue<ValueWithRecordId<OutputT>>>, org.apache.flink.streaming.api.functions.source.SourceFunction<WindowedValue<ValueWithRecordId<OutputT>>>

public class UnboundedSourceWrapper<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>
extends org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<WindowedValue<ValueWithRecordId<OutputT>>>
implements BeamStoppableFunction, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback

Wrapper for executing UnboundedSources as a Flink Source.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Constructor Summary

Constructors

Constructor	Description
UnboundedSourceWrapper(String stepName, PipelineOptions pipelineOptions, UnboundedSource<OutputT,CheckpointMarkT> source, int parallelism)

Method Summary

Modifier and Type	Method	Description
void	cancel()
void	close()
List<? extends UnboundedSource<OutputT,CheckpointMarkT>>	getSplitSources()	Visible so that we can check this in tests.
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	notifyCheckpointComplete(long checkpointId)
void	onProcessingTime(long timestamp)
void	open(org.apache.flink.configuration.Configuration parameters)	Initialize and restore state before starting execution of the source.
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<OutputT>>> ctx)
void	setSourceContext(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<OutputT>>> ctx)	Visible so that we can set this in tests.
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
void	stop()	Unused method for backward compatibility.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Details

UnboundedSourceWrapper

public UnboundedSourceWrapper(String stepName,
 PipelineOptions pipelineOptions,
 UnboundedSource<OutputT,CheckpointMarkT> source,
 int parallelism)
                       throws Exception

Throws:

Exception

Method Details

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Initialize and restore state before starting execution of the source.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<OutputT>>> ctx)
         throws Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OutputT>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OutputT>

stop

public void stop()

Description copied from interface: BeamStoppableFunction

Unused method for backward compatibility.

Specified by:

stop in interface BeamStoppableFunction

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

onProcessingTime

public void onProcessingTime(long timestamp)

Specified by:

onProcessingTime in interface org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback

getSplitSources

public List<? extends UnboundedSource<OutputT,CheckpointMarkT>> getSplitSources()

Visible so that we can check this in tests. Must not be used for anything else.

setSourceContext

public void setSourceContext(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<OutputT>>> ctx)

Visible so that we can set this in tests. This is only set in the run method which is inconvenient for the tests where the context is assumed to be set when run is called. Must not be used for anything else.

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Throws:

Exception