org.apache.beam.sdk.transforms

Class Combine

java.lang.Object

org.apache.beam.sdk.transforms.Combine

public class Combine
extends java.lang.Object

PTransforms for combining PCollection elements globally and per-key.

See the documentation for how to use the operations in this class.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Combine.AccumulatingCombineFn<InputT,AccumT extends Combine.AccumulatingCombineFn.Accumulator<InputT,AccumT,OutputT>,OutputT> A CombineFn that uses a subclass of Combine.AccumulatingCombineFn.Accumulator as its accumulator type.
static class	Combine.BinaryCombineDoubleFn An abstract subclass of Combine.CombineFn for implementing combiners that are more easily and efficiently expressed as binary operations on doubles.
static class	Combine.BinaryCombineFn<V> An abstract subclass of Combine.CombineFn for implementing combiners that are more easily expressed as binary operations.
static class	Combine.BinaryCombineIntegerFn An abstract subclass of Combine.CombineFn for implementing combiners that are more easily and efficiently expressed as binary operations on ints
static class	Combine.BinaryCombineLongFn An abstract subclass of Combine.CombineFn for implementing combiners that are more easily and efficiently expressed as binary operations on longs.
static class	Combine.CombineFn<InputT,AccumT,OutputT> A CombineFn<InputT, AccumT, OutputT> specifies how to combine a collection of input values of type InputT into a single output value of type OutputT.
static class	Combine.Globally<InputT,OutputT> Combine.Globally<InputT, OutputT> takes a PCollection<InputT> and returns a PCollection<OutputT> whose elements are the result of combining all the elements in each window of the input PCollection, using a specified CombineFn<InputT, AccumT, OutputT>.
static class	Combine.GloballyAsSingletonView<InputT,OutputT> Combine.GloballyAsSingletonView<InputT, OutputT> takes a PCollection<InputT> and returns a PCollectionView<OutputT> whose elements are the result of combining all the elements in each window of the input PCollection, using a specified CombineFn<InputT, AccumT, OutputT>.
static class	Combine.GroupedValues<K,InputT,OutputT> GroupedValues<K, InputT, OutputT> takes a PCollection<KV<K, Iterable<InputT>>>, such as the result of GroupByKey, applies a specified CombineFn<InputT, AccumT, OutputT> to each of the input KV<K, Iterable<InputT>> elements to produce a combined output KV<K, OutputT> element, and returns a PCollection<KV<K, OutputT>> containing all the combined output elements.
static class	Combine.Holder<V> Holds a single value value of type V which may or may not be present.
static class	Combine.IterableCombineFn<V> Converts a SerializableFunction from Iterable<V>s to Vs into a simple Combine.CombineFn over Vs.
static class	Combine.PerKey<K,InputT,OutputT> PerKey<K, InputT, OutputT> takes a PCollection<KV<K, InputT>>, groups it by key, applies a combining function to the InputT values associated with each key to produce a combined OutputT value, and returns a PCollection<KV<K, OutputT>> representing a map from each distinct key of the input PCollection to the corresponding combined value.
static class	Combine.PerKeyWithHotKeyFanout<K,InputT,OutputT> Like Combine.PerKey, but sharding the combining of hot keys.
static class	Combine.SimpleCombineFn<V> Deprecated.

Method Summary

Modifier and Type	Method and Description
static <InputT,OutputT> Combine.Globally<InputT,OutputT>	globally(CombineFnBase.GlobalCombineFn<? super InputT,?,OutputT> fn) Returns a Combine.Globally PTransform that uses the given GloballyCombineFn to combine all the elements in each window of the input PCollection into a single value in the output PCollection.
static <V> Combine.Globally<V,V>	globally(SerializableFunction<java.lang.Iterable<V>,V> combiner) Returns a Combine.Globally PTransform that uses the given SerializableFunction to combine all the elements in each window of the input PCollection into a single value in the output PCollection.
static <K,InputT,OutputT> Combine.GroupedValues<K,InputT,OutputT>	groupedValues(CombineFnBase.GlobalCombineFn<? super InputT,?,OutputT> fn) Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given CombineFn to combine all the values associated with a key, ignoring the key.
static <K,V> Combine.GroupedValues<K,V,V>	groupedValues(SerializableFunction<java.lang.Iterable<V>,V> fn) Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given SerializableFunction to combine all the values associated with a key, ignoring the key.
static <K,InputT,OutputT> Combine.PerKey<K,InputT,OutputT>	perKey(CombineFnBase.GlobalCombineFn<? super InputT,?,OutputT> fn) Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the values lists to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.
static <K,V> Combine.PerKey<K,V,V>	perKey(SerializableFunction<java.lang.Iterable<V>,V> fn) Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the values lists to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

globally

public static <V> Combine.Globally<V,V> globally(SerializableFunction<java.lang.Iterable<V>,V> combiner)

Returns a Combine.Globally PTransform that uses the given SerializableFunction to combine all the elements in each window of the input PCollection into a single value in the output PCollection. The types of the input elements and the output elements must be the same.

If the input PCollection is windowed into GlobalWindows, a default value in the GlobalWindow will be output if the input PCollection is empty. To use this with inputs with other windowing, either Combine.Globally.withoutDefaults() or Combine.Globally.asSingletonView() must be called.

See Combine.Globally for more information.

globally

public static <InputT,OutputT> Combine.Globally<InputT,OutputT> globally(CombineFnBase.GlobalCombineFn<? super InputT,?,OutputT> fn)

Returns a Combine.Globally PTransform that uses the given GloballyCombineFn to combine all the elements in each window of the input PCollection into a single value in the output PCollection. The types of the input elements and the output elements can differ.

If the input PCollection is windowed into GlobalWindows, a default value in the GlobalWindow will be output if the input PCollection is empty. To use this with inputs with other windowing, either Combine.Globally.withoutDefaults() or Combine.Globally.asSingletonView() must be called.

See Combine.Globally for more information.

perKey

public static <K,V> Combine.PerKey<K,V,V> perKey(SerializableFunction<java.lang.Iterable<V>,V> fn)

Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the values lists to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.

Each output element is in the window by which its corresponding input was grouped, and has the timestamp of the end of that window. The output PCollection has the same WindowFn as the input.

See Combine.PerKey for more information.

perKey

public static <K,InputT,OutputT> Combine.PerKey<K,InputT,OutputT> perKey(CombineFnBase.GlobalCombineFn<? super InputT,?,OutputT> fn)

Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the values lists to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.

Each output element is in the window by which its corresponding input was grouped, and has the timestamp of the end of that window. The output PCollection has the same WindowFn as the input.

See Combine.PerKey for more information.

groupedValues

public static <K,V> Combine.GroupedValues<K,V,V> groupedValues(SerializableFunction<java.lang.Iterable<V>,V> fn)

Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given SerializableFunction to combine all the values associated with a key, ignoring the key. The type of the input and output values must be the same.

Each output element has the same timestamp and is in the same window as its corresponding input element, and the output PCollection has the same WindowFn associated with it as the input.

See Combine.GroupedValues for more information.

Note that perKey(SerializableFunction) is typically more convenient to use than GroupByKey followed by groupedValues(...).

groupedValues

public static <K,InputT,OutputT> Combine.GroupedValues<K,InputT,OutputT> groupedValues(CombineFnBase.GlobalCombineFn<? super InputT,?,OutputT> fn)

Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given CombineFn to combine all the values associated with a key, ignoring the key. The types of the input and output values can differ.

Each output element has the same timestamp and is in the same window as its corresponding input element, and the output PCollection has the same WindowFn associated with it as the input.

See Combine.GroupedValues for more information.

Note that perKey(CombineFnBase.GlobalCombineFn) is typically more convenient to use than GroupByKey followed by groupedValues(...).