Package org.apache.beam.runners.flink.translation.wrappers.streaming

Class PartialReduceBundleOperator<K,InputT,OutputT,AccumT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<WindowedValue<KV<K,AccumT>>>

org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator<KV<K,InputT>,KV<K,InputT>,KV<K,AccumT>>

org.apache.beam.runners.flink.translation.wrappers.streaming.PartialReduceBundleOperator<K,InputT,OutputT,AccumT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<WindowedValue<KV<K,InputT>>>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<WindowedValue<KV<K,InputT>>,WindowedValue<KV<K,AccumT>>>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<WindowedValue<KV<K,AccumT>>>, org.apache.flink.streaming.api.operators.StreamOperator<WindowedValue<KV<K,AccumT>>>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.Triggerable<FlinkKey,org.apache.beam.runners.core.TimerInternals.TimerData>, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<WindowedValue<KV<K,InputT>>,RawUnionValue,WindowedValue<KV<K,AccumT>>>

public class PartialReduceBundleOperator<K,InputT,OutputT,AccumT>
extends DoFnOperator<KV<K,InputT>,KV<K,InputT>,KV<K,AccumT>>

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

DoFnOperator.BufferedOutputManager<OutputT>, DoFnOperator.FlinkStepContext, DoFnOperator.MultiOutputOutputManagerFactory<OutputT>

Field Summary

Fields inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

additionalOutputTags, bufferingDoFnRunner, doFn, doFnRunner, keyedStateInternals, mainOutputTag, outputManager, outputManagerFactory, pushbackDoFnRunner, serializedOptions, sideInputHandler, sideInputReader, sideInputs, sideInputTagMapping, stepName, timerInternals, timerService, windowingStrategy

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor

Description

PartialReduceBundleOperator(CombineFnBase.GlobalCombineFn<InputT,AccumT,OutputT> combineFn, String stepName, Coder<WindowedValue<KV<K,InputT>>> windowedInputCoder, TupleTag<KV<K,AccumT>> mainOutputTag, List<TupleTag<?>> additionalOutputTags, org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<KV<K,AccumT>> outputManagerFactory, WindowingStrategy<?,?> windowingStrategy, Map<Integer,PCollectionView<?>> sideInputTagMapping, Collection<PCollectionView<?>> sideInputs, PipelineOptions options)

Method Summary

Modifier and Type	Method	Description
protected DoFn<KV<K,InputT>,KV<K,AccumT>>	getDoFn()
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
void	open()
protected boolean	shoudBundleElements()
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

Methods inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

addSideInputValue, applyInputWatermarkHold, applyOutputWatermarkHold, close, createWrappingDoFnRunner, finish, fireTimer, fireTimerInternal, getBundleFinalizer, getCurrentOutputWatermark, getEffectiveInputWatermark, getLockToAcquireForStateAccessDuringBundles, invokeFinishBundle, notifyCheckpointComplete, numProcessingTimeTimers, onEventTime, onProcessingTime, prepareSnapshotPreBarrier, preProcess, processElement, processElement1, processElement2, processWatermark, processWatermark1, processWatermark2, scheduleForCurrentProcessingTime, setBundleFinishedCallback, setPreBundleCallback, setup

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

processLatencyMarker1, processLatencyMarker2, processWatermarkStatus1, processWatermarkStatus2

Constructor Details

PartialReduceBundleOperator

public PartialReduceBundleOperator(CombineFnBase.GlobalCombineFn<InputT,AccumT,OutputT> combineFn,
 String stepName,
 Coder<WindowedValue<KV<K,InputT>>> windowedInputCoder,
 TupleTag<KV<K,AccumT>> mainOutputTag,
 List<TupleTag<?>> additionalOutputTags,
 org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<KV<K,AccumT>> outputManagerFactory,
 WindowingStrategy<?,?> windowingStrategy,
 Map<Integer,PCollectionView<?>> sideInputTagMapping,
 Collection<PCollectionView<?>> sideInputs,
 PipelineOptions options)

Method Details

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<K>

Overrides:

open in class DoFnOperator<KV<K,InputT>,KV<K,InputT>,KV<K,AccumT>>

Throws:

Exception

shoudBundleElements

protected boolean shoudBundleElements()

Overrides:

shoudBundleElements in class DoFnOperator<KV<K,InputT>,KV<K,InputT>,KV<K,AccumT>>

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class DoFnOperator<KV<K,InputT>,KV<K,InputT>,KV<K,AccumT>>

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

snapshotState in class DoFnOperator<KV<K,InputT>,KV<K,InputT>,KV<K,AccumT>>

Throws:

Exception

getDoFn

protected DoFn<KV<K,InputT>,KV<K,AccumT>> getDoFn()

Overrides:

getDoFn in class DoFnOperator<KV<K,InputT>,KV<K,InputT>,KV<K,AccumT>>