Package org.apache.beam.runners.flink.translation.wrappers.streaming.io

Class UnboundedSourceWrapper<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>

java.lang.Object
org.apache.flink.api.common.functions.AbstractRichFunction
org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<WindowedValue<ValueWithRecordId<OutputT>>>
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper<OutputT,CheckpointMarkT>
All Implemented Interfaces:
Serializable, BeamStoppableFunction, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<WindowedValue<ValueWithRecordId<OutputT>>>, org.apache.flink.streaming.api.functions.source.SourceFunction<WindowedValue<ValueWithRecordId<OutputT>>>
public class UnboundedSourceWrapper<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark> extends org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<WindowedValue<ValueWithRecordId<OutputT>>> implements BeamStoppableFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback
Wrapper for executing UnboundedSources as a Flink Source.
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

    org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

  • Constructor Summary

    Constructors
    Constructor
    Description
    UnboundedSourceWrapper(String stepName, PipelineOptions pipelineOptions, UnboundedSource<OutputT,CheckpointMarkT> source, int parallelism)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    cancel()
     
    void
    close()
     
    List<? extends UnboundedSource<OutputT,CheckpointMarkT>>
    getSplitSources()
    Visible so that we can check this in tests.
    void
    initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
     
    void
    notifyCheckpointComplete(long checkpointId)
     
    void
    onProcessingTime(long timestamp)
     
    void
    open(org.apache.flink.configuration.Configuration parameters)
    Initialize and restore state before starting execution of the source.
    void
    run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<OutputT>>> ctx)
     
    void
    setSourceContext(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<OutputT>>> ctx)
    Visible so that we can set this in tests.
    void
    snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
     
    void
    stop()
    Unused method for backward compatibility.

    Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

    getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

    notifyCheckpointAborted

    Methods inherited from interface org.apache.flink.api.common.functions.RichFunction

    open

  • Constructor Details

  • Method Details

    • open

      public void open(org.apache.flink.configuration.Configuration parameters) throws Exception
      Initialize and restore state before starting execution of the source.
      Specified by:
      open in interface org.apache.flink.api.common.functions.RichFunction
      Overrides:
      open in class org.apache.flink.api.common.functions.AbstractRichFunction
      Throws:
      Exception

    • run

      public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<OutputT>>> ctx) throws Exception
      Specified by:
      run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OutputT>
      Throws:
      Exception

    • close

      public void close() throws Exception
      Specified by:
      close in interface org.apache.flink.api.common.functions.RichFunction
      Overrides:
      close in class org.apache.flink.api.common.functions.AbstractRichFunction
      Throws:
      Exception

    • cancel

      public void cancel()
      Specified by:
      cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OutputT>

    • stop

      public void stop()
      Description copied from interface: BeamStoppableFunction
      Unused method for backward compatibility.
      Specified by:
      stop in interface BeamStoppableFunction

    • snapshotState

      public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext) throws Exception
      Specified by:
      snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
      Throws:
      Exception

    • initializeState

      public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context) throws Exception
      Specified by:
      initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
      Throws:
      Exception

    • onProcessingTime

      public void onProcessingTime(long timestamp)
      Specified by:
      onProcessingTime in interface org.apache.flink.api.common.operators.ProcessingTimeService.ProcessingTimeCallback

    • getSplitSources

      public List<? extends UnboundedSource<OutputT,CheckpointMarkT>> getSplitSources()
      Visible so that we can check this in tests. Must not be used for anything else.

    • setSourceContext

      public void setSourceContext(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<OutputT>>> ctx)
      Visible so that we can set this in tests. This is only set in the run method which is inconvenient for the tests where the context is assumed to be set when run is called. Must not be used for anything else.

    • notifyCheckpointComplete

      public void notifyCheckpointComplete(long checkpointId) throws Exception
      Specified by:
      notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener
      Throws:
      Exception