Package org.apache.beam.runners.flink.translation.functions

Class FlinkReduceFunction<K,AccumT,OutputT,W extends BoundedWindow>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.api.common.functions.RichGroupReduceFunction<WindowedValue<KV<K,AccumT>>,WindowedValue<KV<K,OutputT>>>

org.apache.beam.runners.flink.translation.functions.FlinkReduceFunction<K,AccumT,OutputT,W>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.GroupReduceFunction<WindowedValue<KV<K,AccumT>>,WindowedValue<KV<K,OutputT>>>, org.apache.flink.api.common.functions.RichFunction

public class FlinkReduceFunction<K,AccumT,OutputT,W extends BoundedWindow>
extends org.apache.flink.api.common.functions.RichGroupReduceFunction<WindowedValue<KV<K,AccumT>>,WindowedValue<KV<K,OutputT>>>

This is the second part for executing a Combine.PerKey on Flink, the second part is FlinkReduceFunction. This function performs the final combination of the pre-combined values after a shuffle.

The input to reduce(Iterable, Collector) are elements of the same key but for different windows. We have to ensure that we only combine elements of matching windows.

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected final CombineFnBase.GlobalCombineFn<?,AccumT,OutputT>	combineFn
protected final org.apache.beam.runners.core.construction.SerializablePipelineOptions	serializedOptions
protected final Map<PCollectionView<?>,WindowingStrategy<?,?>>	sideInputs
protected final WindowingStrategy<Object,W>	windowingStrategy

Constructor Summary

Constructors

Constructor	Description
FlinkReduceFunction(CombineFnBase.GlobalCombineFn<?,AccumT,OutputT> combineFn, WindowingStrategy<Object,W> windowingStrategy, Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs, PipelineOptions pipelineOptions)
FlinkReduceFunction(CombineFnBase.GlobalCombineFn<?,AccumT,OutputT> combineFn, WindowingStrategy<Object,W> windowingStrategy, Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs, PipelineOptions pipelineOptions, boolean groupedByWindow)

Method Summary

Modifier and Type	Method	Description
void	open(org.apache.flink.configuration.Configuration parameters)
void	reduce(Iterable<WindowedValue<KV<K,AccumT>>> elements, org.apache.flink.util.Collector<WindowedValue<KV<K,OutputT>>> out)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

combineFn

protected final CombineFnBase.GlobalCombineFn<?,AccumT,OutputT> combineFn

windowingStrategy

protected final WindowingStrategy<Object,W extends BoundedWindow> windowingStrategy

sideInputs

protected final Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs

serializedOptions

protected final org.apache.beam.runners.core.construction.SerializablePipelineOptions serializedOptions

Constructor Details

FlinkReduceFunction

public FlinkReduceFunction(CombineFnBase.GlobalCombineFn<?,AccumT,OutputT> combineFn,
 WindowingStrategy<Object,W> windowingStrategy,
 Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs,
 PipelineOptions pipelineOptions)

FlinkReduceFunction

public FlinkReduceFunction(CombineFnBase.GlobalCombineFn<?,AccumT,OutputT> combineFn,
 WindowingStrategy<Object,W> windowingStrategy,
 Map<PCollectionView<?>,WindowingStrategy<?,?>> sideInputs,
 PipelineOptions pipelineOptions,
 boolean groupedByWindow)

Method Details

open

public void open(org.apache.flink.configuration.Configuration parameters)

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

reduce

public void reduce(Iterable<WindowedValue<KV<K,AccumT>>> elements,
 org.apache.flink.util.Collector<WindowedValue<KV<K,OutputT>>> out)
            throws Exception

Specified by:

reduce in interface org.apache.flink.api.common.functions.GroupReduceFunction<K,AccumT>

Specified by:

reduce in class org.apache.flink.api.common.functions.RichGroupReduceFunction<WindowedValue<KV<K,AccumT>>,WindowedValue<KV<K,OutputT>>>

Throws:

Exception