Package org.apache.beam.runners.flink.translation.wrappers.streaming.io

Class DedupingOperator<T>

java.lang.Object
org.apache.flink.streaming.api.operators.AbstractStreamOperator<WindowedValue<T>>
org.apache.beam.runners.flink.translation.wrappers.streaming.io.DedupingOperator<T>
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<WindowedValue<ValueWithRecordId<T>>>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<WindowedValue<ValueWithRecordId<T>>,WindowedValue<T>>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<WindowedValue<T>>, org.apache.flink.streaming.api.operators.StreamOperator<WindowedValue<T>>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.Triggerable<FlinkKey,org.apache.flink.runtime.state.VoidNamespace>
public class DedupingOperator<T> extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<WindowedValue<T>> implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<WindowedValue<ValueWithRecordId<T>>,WindowedValue<T>>, org.apache.flink.streaming.api.operators.Triggerable<FlinkKey,org.apache.flink.runtime.state.VoidNamespace>
Remove values with duplicate ids.
See Also:

  • Field Summary

    Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    chainingStrategy, config, latencyStats, LOG, metrics, output, processingTimeService

  • Constructor Summary

    Constructors
    Constructor
    Description
    DedupingOperator(PipelineOptions options)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
     
    void
    onEventTime(org.apache.flink.streaming.api.operators.InternalTimer<FlinkKey,org.apache.flink.runtime.state.VoidNamespace> internalTimer)
     
    void
    onProcessingTime(org.apache.flink.streaming.api.operators.InternalTimer<FlinkKey,org.apache.flink.runtime.state.VoidNamespace> internalTimer)
     
    void
    open()
     
    void
    processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<WindowedValue<ValueWithRecordId<T>>> streamRecord)
     

    Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

    close, finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

    notifyCheckpointAborted, notifyCheckpointComplete

    Methods inherited from interface org.apache.flink.streaming.api.operators.Input

    processLatencyMarker, processWatermark, processWatermarkStatus

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

    getCurrentKey, setCurrentKey

    Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

    hasKeyContext

    Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

    setKeyContextElement

    Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

    close, finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

  • Constructor Details

  • Method Details

    • initializeState

      public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context) throws Exception
      Specified by:
      initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
      Overrides:
      initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<WindowedValue<T>>
      Throws:
      Exception

    • open

      public void open()
      Specified by:
      open in interface org.apache.flink.streaming.api.operators.StreamOperator<T>
      Overrides:
      open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<WindowedValue<T>>

    • processElement

      public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<WindowedValue<ValueWithRecordId<T>>> streamRecord) throws Exception
      Specified by:
      processElement in interface org.apache.flink.streaming.api.operators.Input<T>
      Throws:
      Exception

    • onEventTime

      public void onEventTime(org.apache.flink.streaming.api.operators.InternalTimer<FlinkKey,org.apache.flink.runtime.state.VoidNamespace> internalTimer)
      Specified by:
      onEventTime in interface org.apache.flink.streaming.api.operators.Triggerable<FlinkKey,org.apache.flink.runtime.state.VoidNamespace>

    • onProcessingTime

      public void onProcessingTime(org.apache.flink.streaming.api.operators.InternalTimer<FlinkKey,org.apache.flink.runtime.state.VoidNamespace> internalTimer) throws Exception
      Specified by:
      onProcessingTime in interface org.apache.flink.streaming.api.operators.Triggerable<FlinkKey,org.apache.flink.runtime.state.VoidNamespace>
      Throws:
      Exception