Package org.apache.beam.runners.flink

Class FlinkStreamingAggregationsTranslators

java.lang.Object
org.apache.beam.runners.flink.FlinkStreamingAggregationsTranslators
public class FlinkStreamingAggregationsTranslators extends Object

  • Constructor Details

    • FlinkStreamingAggregationsTranslators

      public FlinkStreamingAggregationsTranslators()

  • Method Details

    • getWindowedAggregateDoFnOperator

      public static <K, InputAccumT, OutputAccumT, InputT, OutputT> WindowDoFnOperator<K,InputAccumT,OutputAccumT> getWindowedAggregateDoFnOperator(org.apache.beam.runners.flink.FlinkStreamingTranslationContext context, PTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>> transform, KvCoder<K,InputAccumT> inputKvCoder, Coder<WindowedValue<KV<K,OutputAccumT>>> outputCoder, org.apache.beam.runners.core.SystemReduceFn<K,InputAccumT,?,OutputAccumT,BoundedWindow> reduceFn, Map<Integer,PCollectionView<?>> sideInputTagMapping, List<PCollectionView<?>> sideInputs)
      Create a DoFnOperator instance that group elements per window and apply a combine function on them.

    • getWindowedAggregateDoFnOperator

      public static <K, InputAccumT, OutputAccumT, InputT, OutputT> WindowDoFnOperator<K,InputAccumT,OutputAccumT> getWindowedAggregateDoFnOperator(org.apache.beam.runners.flink.FlinkStreamingTranslationContext context, PTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>> transform, KvCoder<K,InputAccumT> inputKvCoder, Coder<WindowedValue<KV<K,OutputAccumT>>> outputCoder, CombineFnBase.GlobalCombineFn<? super InputAccumT,?,OutputAccumT> combineFn, Map<Integer,PCollectionView<?>> sideInputTagMapping, List<PCollectionView<?>> sideInputs)

    • getBatchCombinePerKeyOperator

      public static <K, InputT, AccumT, OutputT> org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<WindowedValue<KV<K,OutputT>>> getBatchCombinePerKeyOperator(org.apache.beam.runners.flink.FlinkStreamingTranslationContext context, PCollection<KV<K,InputT>> input, Map<Integer,PCollectionView<?>> sideInputTagMapping, List<PCollectionView<?>> sideInputs, Coder<WindowedValue<KV<K,AccumT>>> windowedAccumCoder, CombineFnBase.GlobalCombineFn<InputT,AccumT,?> combineFn, WindowDoFnOperator<K,AccumT,OutputT> finalDoFnOperator, org.apache.flink.api.common.typeinfo.TypeInformation<WindowedValue<KV<K,OutputT>>> outputTypeInfo)

    • batchGroupByKey

      public static <K, InputT> org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<WindowedValue<KV<K,Iterable<InputT>>>> batchGroupByKey(org.apache.beam.runners.flink.FlinkStreamingTranslationContext context, PTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,Iterable<InputT>>>> transform)
      Creates a two-steps GBK operation. Elements are first aggregated locally to save on serialized size since in batch it's very likely that all the elements will be within the same window and pane. The only difference with batchCombinePerKey is the nature of the SystemReduceFn used. It uses SystemReduceFn.buffering() instead of SystemReduceFn.combining() so that new element can simply be appended without accessing the existing state.

    • batchCombinePerKey

      public static <K, InputT, AccumT, OutputT> org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<WindowedValue<KV<K,OutputT>>> batchCombinePerKey(org.apache.beam.runners.flink.FlinkStreamingTranslationContext context, PTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>> transform, CombineFnBase.GlobalCombineFn<InputT,AccumT,OutputT> combineFn, Map<Integer,PCollectionView<?>> sideInputTagMapping, List<PCollectionView<?>> sideInputs)

    • buildTwoInputStream

      public static <K, InputT, OutputT> org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<WindowedValue<KV<K,OutputT>>> buildTwoInputStream(org.apache.flink.streaming.api.datastream.KeyedStream<WindowedValue<KV<K,InputT>>,FlinkKey> keyedStream, org.apache.flink.streaming.api.datastream.DataStream<RawUnionValue> sideInputStream, String name, WindowDoFnOperator<K,InputT,OutputT> operator, org.apache.flink.api.common.typeinfo.TypeInformation<WindowedValue<KV<K,OutputT>>> outputTypeInfo)

    • batchCombinePerKeyNoSideInputs

      public static <K, InputT, AccumT, OutputT> org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<WindowedValue<KV<K,OutputT>>> batchCombinePerKeyNoSideInputs(org.apache.beam.runners.flink.FlinkStreamingTranslationContext context, PTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>> transform, CombineFnBase.GlobalCombineFn<InputT,AccumT,OutputT> combineFn)