Package org.apache.beam.runners.flink.translation.wrappers.streaming.stableinput

Class BufferingDoFnRunner<InputT,OutputT>

java.lang.Object

org.apache.beam.runners.flink.translation.wrappers.streaming.stableinput.BufferingDoFnRunner<InputT,OutputT>

All Implemented Interfaces:

org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

public class BufferingDoFnRunner<InputT,OutputT>
extends Object
implements org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

A DoFnRunner which buffers data for supporting DoFn.RequiresStableInput.

When a DoFn is annotated with @RequiresStableInput we are only allowed to process elements after a checkpoint has completed. This ensures that the input is stable and we produce idempotent results on failures.

Method Summary

Modifier and Type	Method	Description
void	checkpoint(long checkpointId)	Should be called when a checkpoint is created.
void	checkpointCompleted(long checkpointId)	Should be called when a checkpoint is completed.
static <InputT, OutputT> BufferingDoFnRunner<InputT,OutputT>	create(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> doFnRunner, String stateName, Coder windowedInputCoder, Coder windowCoder, org.apache.flink.runtime.state.OperatorStateBackend operatorStateBackend, @Nullable org.apache.flink.runtime.state.KeyedStateBackend<Object> keyedStateBackend, int maxConcurrentCheckpoints, org.apache.beam.runners.core.construction.SerializablePipelineOptions pipelineOptions)
static <InputT, OutputT> BufferingDoFnRunner<InputT,OutputT>	create(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> doFnRunner, String stateName, Coder windowedInputCoder, Coder windowCoder, org.apache.flink.runtime.state.OperatorStateBackend operatorStateBackend, @Nullable org.apache.flink.runtime.state.KeyedStateBackend<Object> keyedStateBackend, int maxConcurrentCheckpoints, org.apache.beam.runners.core.construction.SerializablePipelineOptions pipelineOptions, @Nullable Supplier<Locker> locker, @Nullable Function<InputT,Object> keySelector, @Nullable Runnable finishBundleCallback)
void	finishBundle()
DoFn<InputT,OutputT>	getFn()
long	getOutputWatermarkHold()
<KeyT> void	onTimer(String timerId, String timerFamilyId, KeyT key, BoundedWindow window, Instant timestamp, Instant outputTimestamp, TimeDomain timeDomain)
<KeyT> void	onWindowExpiration(BoundedWindow window, Instant timestamp, KeyT key)
void	processElement(WindowedValue<InputT> elem)
void	startBundle()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

create

public static <InputT,
OutputT>
BufferingDoFnRunner<InputT,OutputT> create(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> doFnRunner,
 String stateName,
 Coder windowedInputCoder,
 Coder windowCoder,
 org.apache.flink.runtime.state.OperatorStateBackend operatorStateBackend,
 @Nullable org.apache.flink.runtime.state.KeyedStateBackend<Object> keyedStateBackend,
 int maxConcurrentCheckpoints,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions pipelineOptions)
                                    throws Exception

Throws:

Exception

create

public static <InputT,
OutputT>
BufferingDoFnRunner<InputT,OutputT> create(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> doFnRunner,
 String stateName,
 Coder windowedInputCoder,
 Coder windowCoder,
 org.apache.flink.runtime.state.OperatorStateBackend operatorStateBackend,
 @Nullable org.apache.flink.runtime.state.KeyedStateBackend<Object> keyedStateBackend,
 int maxConcurrentCheckpoints,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions pipelineOptions,
 @Nullable Supplier<Locker> locker,
 @Nullable Function<InputT,Object> keySelector,
 @Nullable Runnable finishBundleCallback)
                                    throws Exception

Throws:

Exception

startBundle

public void startBundle()

Specified by:

startBundle in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

processElement

public void processElement(WindowedValue<InputT> elem)

Specified by:

processElement in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

onTimer

public <KeyT> void onTimer(String timerId,
 String timerFamilyId,
 KeyT key,
 BoundedWindow window,
 Instant timestamp,
 Instant outputTimestamp,
 TimeDomain timeDomain)

Specified by:

onTimer in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

finishBundle

public void finishBundle()

Specified by:

finishBundle in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

onWindowExpiration

public <KeyT> void onWindowExpiration(BoundedWindow window,
 Instant timestamp,
 KeyT key)

Specified by:

onWindowExpiration in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

getFn

public DoFn<InputT,OutputT> getFn()

Specified by:

getFn in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

checkpoint

public void checkpoint(long checkpointId)
                throws Exception

Should be called when a checkpoint is created.

Throws:

Exception

checkpointCompleted

public void checkpointCompleted(long checkpointId)
                         throws Exception

Should be called when a checkpoint is completed.

Throws:

Exception

getOutputWatermarkHold

public long getOutputWatermarkHold()