Package org.apache.beam.runners.flink.translation.wrappers.streaming

Class DoFnOperator.BufferedOutputManager<OutputT>

java.lang.Object
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.BufferedOutputManager<OutputT>
All Implemented Interfaces:
org.apache.beam.sdk.util.WindowedValueMultiReceiver
Enclosing class:
DoFnOperator<PreInputT,InputT,OutputT>
public static class DoFnOperator.BufferedOutputManager<OutputT> extends Object implements org.apache.beam.sdk.util.WindowedValueMultiReceiver
A WindowedValueReceiver that can buffer its outputs. Uses PushedBackElementsHandler to buffer the data. Buffering data is necessary because no elements can be emitted during snapshotState which is called when the checkpoint barrier already has been sent downstream. Emitting elements would break the flow of checkpoint barrier and violate exactly-once semantics.

This buffering can be deactived using FlinkPipelineOptions#setFinishBundleBeforeCheckpointing(true). If activated, we flush out bundle data before the barrier is sent downstream. This is done via prepareSnapshotPreBarrier. When Flink supports unaligned checkpoints, this should become the default and this class should be removed as in https://github.com/apache/beam/pull/9652.

  • Field Details

    • output

      protected final org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<WindowedValue<OutputT>>> output

  • Method Details

    • output

      public <T> void output(TupleTag<T> tag, WindowedValue<T> value)
      Specified by:
      output in interface org.apache.beam.sdk.util.WindowedValueMultiReceiver