Package org.apache.beam.runners.flink.translation.wrappers.streaming

Class SplittableDoFnOperator<InputT,OutputT,RestrictionT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<WindowedValue<OutputT>>

org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT>

org.apache.beam.runners.flink.translation.wrappers.streaming.SplittableDoFnOperator<InputT,OutputT,RestrictionT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>>>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>>,WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.StreamOperator<WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.Triggerable<FlinkKey,org.apache.beam.runners.core.TimerInternals.TimerData>, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>>,RawUnionValue,WindowedValue<OutputT>>

public class SplittableDoFnOperator<InputT,OutputT,RestrictionT>
extends DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT>

Flink operator for executing splittable DoFns. Specifically, for executing the @ProcessElement method of a splittable DoFn.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

DoFnOperator.BufferedOutputManager<OutputT>, DoFnOperator.FlinkStepContext, DoFnOperator.MultiOutputOutputManagerFactory<OutputT>

Field Summary

Fields inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

additionalOutputTags, bufferingDoFnRunner, doFn, doFnRunner, keyedStateInternals, mainOutputTag, outputManager, outputManagerFactory, pushbackDoFnRunner, serializedOptions, sideInputHandler, sideInputReader, sideInputs, sideInputTagMapping, stepName, timerInternals, timerService, windowingStrategy

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor

Description

SplittableDoFnOperator(DoFn<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT> doFn, String stepName, Coder<WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>>> windowedInputCoder, Map<TupleTag<?>,Coder<?>> outputCoders, TupleTag<OutputT> mainOutputTag, List<TupleTag<?>> additionalOutputTags, org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory, WindowingStrategy<?,?> windowingStrategy, Map<Integer,PCollectionView<?>> sideInputTagMapping, Collection<PCollectionView<?>> sideInputs, PipelineOptions options, Coder<?> keyCoder, org.apache.flink.api.java.functions.KeySelector<WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>>,?> keySelector)

Method Summary

Modifier and Type	Method	Description
void	close()
protected org.apache.beam.runners.core.DoFnRunner<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT>	createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT> wrappedRunner, org.apache.beam.runners.core.StepContext stepContext)
protected void	fireTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)

Methods inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

addSideInputValue, applyInputWatermarkHold, applyOutputWatermarkHold, finish, fireTimerInternal, getBundleFinalizer, getCurrentOutputWatermark, getDoFn, getEffectiveInputWatermark, getLockToAcquireForStateAccessDuringBundles, invokeFinishBundle, notifyCheckpointComplete, numProcessingTimeTimers, onEventTime, onProcessingTime, open, prepareSnapshotPreBarrier, preProcess, processElement, processElement1, processElement2, processWatermark, processWatermark1, processWatermark2, scheduleForCurrentProcessingTime, setBundleFinishedCallback, setPreBundleCallback, setup, shoudBundleElements, snapshotState

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

processLatencyMarker1, processLatencyMarker2, processWatermarkStatus1, processWatermarkStatus2

Constructor Details

SplittableDoFnOperator

public SplittableDoFnOperator(DoFn<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT> doFn,
 String stepName,
 Coder<WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>>> windowedInputCoder,
 Map<TupleTag<?>,Coder<?>> outputCoders,
 TupleTag<OutputT> mainOutputTag,
 List<TupleTag<?>> additionalOutputTags,
 org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory,
 WindowingStrategy<?,?> windowingStrategy,
 Map<Integer,PCollectionView<?>> sideInputTagMapping,
 Collection<PCollectionView<?>> sideInputs,
 PipelineOptions options,
 Coder<?> keyCoder,
 org.apache.flink.api.java.functions.KeySelector<WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>>,?> keySelector)

Method Details

createWrappingDoFnRunner

protected org.apache.beam.runners.core.DoFnRunner<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT> createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT> wrappedRunner,
 org.apache.beam.runners.core.StepContext stepContext)

Overrides:

createWrappingDoFnRunner in class DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT>

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT>

Throws:

Exception

fireTimer

protected void fireTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)

Overrides:

fireTimer in class DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT>

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.streaming.api.operators.StreamOperator<InputT>

Overrides:

close in class DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,org.apache.beam.runners.core.KeyedWorkItem<byte[],KV<InputT,RestrictionT>>,OutputT>

Throws:

Exception