org.apache.beam.runners.spark.stateful

Class SparkGroupAlsoByWindowViaWindowSet

java.lang.Object

org.apache.beam.runners.spark.stateful.SparkGroupAlsoByWindowViaWindowSet

public class SparkGroupAlsoByWindowViaWindowSet
extends java.lang.Object

An implementation of GroupAlsoByWindowViaWindowSetDoFn logic for grouping by windows and controlling trigger firings and pane accumulation.

This implementation is a composite of Spark transformations revolving around state management using Spark's PairDStreamFunctions.updateStateByKey(Function1, Partitioner, boolean, ClassTag) to update state with new data and timers.

Using updateStateByKey allows to scan through the entire state visiting not just the updated state (new values for key) but also check if timers are ready to fire. Since updateStateByKey bounds the types of state and output to be the same, a (state, output) tuple is used, filtering the state (and output if no firing) in the following steps.

Constructor Summary

Constructors

Constructor and Description
SparkGroupAlsoByWindowViaWindowSet()

Method Summary

Modifier and Type

Method and Description

static <K,InputT,W extends BoundedWindow> org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<KV<K,java.lang.Iterable<InputT>>>>

groupAlsoByWindow(org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<KV<K,java.lang.Iterable<org.apache.beam.sdk.util.WindowedValue<InputT>>>>> inputDStream, Coder<K> keyCoder, Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> wvCoder, WindowingStrategy<?,W> windowingStrategy, org.apache.beam.runners.spark.translation.SparkRuntimeContext runtimeContext, java.util.List<java.lang.Integer> sourceIds)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkGroupAlsoByWindowViaWindowSet

public SparkGroupAlsoByWindowViaWindowSet()

Method Detail

groupAlsoByWindow

public static <K,InputT,W extends BoundedWindow> org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<KV<K,java.lang.Iterable<InputT>>>> groupAlsoByWindow(org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<KV<K,java.lang.Iterable<org.apache.beam.sdk.util.WindowedValue<InputT>>>>> inputDStream,
                                                                                                                                                                                             Coder<K> keyCoder,
                                                                                                                                                                                             Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> wvCoder,
                                                                                                                                                                                             WindowingStrategy<?,W> windowingStrategy,
                                                                                                                                                                                             org.apache.beam.runners.spark.translation.SparkRuntimeContext runtimeContext,
                                                                                                                                                                                             java.util.List<java.lang.Integer> sourceIds)