org.apache.beam.runners.spark.structuredstreaming.translation.batch

Class Aggregators

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.batch.Aggregators

public class Aggregators
extends java.lang.Object

Constructor Summary

Constructors

Constructor and Description
Aggregators()

Method Summary

Modifier and Type	Method and Description
static <ValT,AccT,ResT,InT> org.apache.spark.sql.expressions.Aggregator<InT,?,ResT>	value(Combine.CombineFn<ValT,AccT,ResT> fn, ScalaInterop.Fun1<InT,ValT> valueFn, org.apache.spark.sql.Encoder<AccT> accEnc, org.apache.spark.sql.Encoder<ResT> outEnc) Creates simple value Aggregator that is not window aware.
static <ValT,AccT,ResT,InT> org.apache.spark.sql.expressions.Aggregator<org.apache.beam.sdk.util.WindowedValue<InT>,?,java.util.Collection<org.apache.beam.sdk.util.WindowedValue<ResT>>>	windowedValue(Combine.CombineFn<ValT,AccT,ResT> fn, ScalaInterop.Fun1<org.apache.beam.sdk.util.WindowedValue<InT>,ValT> valueFn, WindowingStrategy<?,?> windowing, org.apache.spark.sql.Encoder<BoundedWindow> windowEnc, org.apache.spark.sql.Encoder<AccT> accEnc, org.apache.spark.sql.Encoder<org.apache.beam.sdk.util.WindowedValue<ResT>> outEnc) Creates windowed Spark Aggregator depending on the provided Beam WindowFns.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Aggregators

public Aggregators()

Method Detail

value

public static <ValT,AccT,ResT,InT> org.apache.spark.sql.expressions.Aggregator<InT,?,ResT> value(Combine.CombineFn<ValT,AccT,ResT> fn,
                                                                                                 ScalaInterop.Fun1<InT,ValT> valueFn,
                                                                                                 org.apache.spark.sql.Encoder<AccT> accEnc,
                                                                                                 org.apache.spark.sql.Encoder<ResT> outEnc)

Creates simple value Aggregator that is not window aware.

Type Parameters:

ValT - Combine.CombineFn input type

AccT - Combine.CombineFn accumulator type

ResT - Combine.CombineFn / Aggregator result type

InT - Aggregator input type

windowedValue

public static <ValT,AccT,ResT,InT> org.apache.spark.sql.expressions.Aggregator<org.apache.beam.sdk.util.WindowedValue<InT>,?,java.util.Collection<org.apache.beam.sdk.util.WindowedValue<ResT>>> windowedValue(Combine.CombineFn<ValT,AccT,ResT> fn,
                                                                                                                                                                                                               ScalaInterop.Fun1<org.apache.beam.sdk.util.WindowedValue<InT>,ValT> valueFn,
                                                                                                                                                                                                               WindowingStrategy<?,?> windowing,
                                                                                                                                                                                                               org.apache.spark.sql.Encoder<BoundedWindow> windowEnc,
                                                                                                                                                                                                               org.apache.spark.sql.Encoder<AccT> accEnc,
                                                                                                                                                                                                               org.apache.spark.sql.Encoder<org.apache.beam.sdk.util.WindowedValue<ResT>> outEnc)

Creates windowed Spark Aggregator depending on the provided Beam WindowFns.

Specialised implementations are provided for:

Sessions Non merging window functions Merging window functions

Type Parameters:

ValT - Combine.CombineFn input type

AccT - Combine.CombineFn accumulator type

ResT - Combine.CombineFn / Aggregator result type

InT - Aggregator input type