org.apache.beam.sdk.transforms

Class Reshuffle<K,V>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<KV<K,V>>,PCollection<KV<K,V>>>

org.apache.beam.sdk.transforms.Reshuffle<K,V>

Type Parameters:

K - The type of key being reshuffled on.

V - The type of value being reshuffled.

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

@Internal
public class Reshuffle<K,V>
extends PTransform<PCollection<KV<K,V>>,PCollection<KV<K,V>>>

For internal use only; no backwards compatibility guarantees.

A PTransform that returns a PCollection equivalent to its input but operationally provides some of the side effects of a GroupByKey, in particular redistribution of elements between workers, checkpointing, and preventing fusion of the surrounding transforms. Some of these side effects (e.g. checkpointing) are not portable and are not guaranteed to occur on all runners.

Performs a GroupByKey so that the data is key-partitioned. Configures the WindowingStrategy so that no data is dropped, but doesn't affect the need for the user to specify allowed lateness and accumulation mode before a user-inserted GroupByKey.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Reshuffle.AssignShardFn<T>
static class	Reshuffle.ViaRandomKey<T> Implementation of viaRandomKey().

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Method Summary

Modifier and Type	Method and Description
PCollection<KV<K,V>>	expand(PCollection<KV<K,V>> input) Override this method to specify how this PTransform should be expanded on the given InputT.
static <K,V> Reshuffle<K,V>	of()
static <T> Reshuffle.ViaRandomKey<T>	viaRandomKey() Encapsulates the sequence "pair input with unique key, apply of(), drop the key" commonly used to break fusion.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

of

public static <K,V> Reshuffle<K,V> of()

viaRandomKey

public static <T> Reshuffle.ViaRandomKey<T> viaRandomKey()

Encapsulates the sequence "pair input with unique key, apply of(), drop the key" commonly used to break fusion.

expand

public PCollection<KV<K,V>> expand(PCollection<KV<K,V>> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<KV<K,V>>,PCollection<KV<K,V>>>