Package org.apache.beam.runners.spark.stateful

Class SparkGroupAlsoByWindowViaWindowSet

java.lang.Object

org.apache.beam.runners.spark.stateful.SparkGroupAlsoByWindowViaWindowSet

All Implemented Interfaces:

Serializable

public class SparkGroupAlsoByWindowViaWindowSet
extends Object
implements Serializable

An implementation of GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow logic for grouping by windows and controlling trigger firings and pane accumulation.

This implementation is a composite of Spark transformations revolving around state management using Spark's PairDStreamFunctions.updateStateByKey(scala.Function1, org.apache.spark.Partitioner, boolean, scala.reflect.ClassTag) to update state with new data and timers.

Using updateStateByKey allows to scan through the entire state visiting not just the updated state (new values for key) but also check if timers are ready to fire. Since updateStateByKey bounds the types of state and output to be the same, a (state, output) tuple is used, filtering the state (and output if no firing) in the following steps.

See Also:

• Serialized Form

Constructor Summary

Constructors

Constructor	Description
SparkGroupAlsoByWindowViaWindowSet()

Method Summary

Modifier and Type	Method	Description
static <K, InputT, W extends BoundedWindow> org.apache.spark.streaming.api.java.JavaDStream<WindowedValue<KV<K,Iterable<InputT>>>>	groupByKeyAndWindow(org.apache.spark.streaming.api.java.JavaDStream<WindowedValue<KV<K,InputT>>> inputDStream, Coder<K> keyCoder, Coder<WindowedValue<InputT>> wvCoder, WindowingStrategy<?,W> windowingStrategy, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, List<Integer> sourceIds, String transformFullName)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SparkGroupAlsoByWindowViaWindowSet

public SparkGroupAlsoByWindowViaWindowSet()

Method Details

groupByKeyAndWindow

public static <K,
InputT,
W extends BoundedWindow>
org.apache.spark.streaming.api.java.JavaDStream<WindowedValue<KV<K,Iterable<InputT>>>> groupByKeyAndWindow(org.apache.spark.streaming.api.java.JavaDStream<WindowedValue<KV<K,InputT>>> inputDStream,
 Coder<K> keyCoder,
 Coder<WindowedValue<InputT>> wvCoder,
 WindowingStrategy<?,W> windowingStrategy,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions options,
 List<Integer> sourceIds,
 String transformFullName)