Package org.apache.beam.runners.spark.translation

Class SparkCombineFn<InputT,ValueT,AccumT,OutputT>

java.lang.Object

org.apache.beam.runners.spark.translation.SparkCombineFn<InputT,ValueT,AccumT,OutputT>

All Implemented Interfaces:

Serializable

public class SparkCombineFn<InputT,ValueT,AccumT,OutputT>
extends Object
implements Serializable

A CombineFnBase.GlobalCombineFn with a CombineWithContext.Context for the SparkRunner.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	SparkCombineFn.WindowedAccumulator<InputT,ValueT,AccumT,ImplT extends SparkCombineFn.WindowedAccumulator<InputT,ValueT,AccumT,ImplT>>	Accumulator of WindowedValues holding values for different windows.

Method Summary

Modifier and Type	Method	Description
Stream<WindowedValue<OutputT>>	extractOutputStream(SparkCombineFn.WindowedAccumulator<?,?,AccumT,?> accumulator)	Extracts the stream of accumulated values.
static <InputT, AccumT, OutputT> SparkCombineFn<InputT,InputT,AccumT,OutputT>	globally(CombineWithContext.CombineFnWithContext<InputT,AccumT,OutputT> combineFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, WindowingStrategy<?,?> windowingStrategy)
static <K, V, AccumT, OutputT> SparkCombineFn<KV<K,V>,V,AccumT,OutputT>	keyed(CombineWithContext.CombineFnWithContext<V,AccumT,OutputT> combineFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, WindowingStrategy<?,?> windowingStrategy)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

keyed

public static <K,
V,
AccumT,
OutputT>
SparkCombineFn<KV<K,V>,V,AccumT,OutputT> keyed(CombineWithContext.CombineFnWithContext<V,AccumT,OutputT> combineFn,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions options,
 Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs,
 WindowingStrategy<?,?> windowingStrategy)

globally

public static <InputT,
AccumT,
OutputT>
SparkCombineFn<InputT,InputT,AccumT,OutputT> globally(CombineWithContext.CombineFnWithContext<InputT,AccumT,OutputT> combineFn,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions options,
 Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs,
 WindowingStrategy<?,?> windowingStrategy)

extractOutputStream

public Stream<WindowedValue<OutputT>> extractOutputStream(SparkCombineFn.WindowedAccumulator<?,?,AccumT,?> accumulator)

Extracts the stream of accumulated values.