Package org.apache.beam.sdk.transforms

Class Flatten

java.lang.Object

org.apache.beam.sdk.transforms.Flatten

public class Flatten
extends Object

Flatten<T> takes multiple PCollection<T>s bundled into a PCollectionList<T> and returns a single PCollection<T> containing all the elements in all the input PCollections. The name "Flatten" suggests taking a list of lists and flattening them into a single list.

Example of use:

 PCollection<String> pc1 = ...;
 PCollection<String> pc2 = ...;
 PCollection<String> pc3 = ...;
 PCollectionList<String> pcs = PCollectionList.of(pc1).and(pc2).and(pc3);
 PCollection<String> merged = pcs.apply(Flatten.<String>pCollections());
 

By default, the Coder of the output PCollection is the same as the Coder of the first PCollection in the input PCollectionList (if the PCollectionList is non-empty).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Flatten.Iterables<T>	FlattenIterables<T> takes a PCollection<Iterable<T>> and returns a PCollection<T> that contains all the elements from each iterable.
static class	Flatten.PCollections<T>	A PTransform that flattens a PCollectionList into a PCollection containing all the elements of all the PCollections in its input.

Constructor Summary

Constructors

Constructor	Description
Flatten()

Method Summary

Modifier and Type	Method	Description
static <T> Flatten.Iterables<T>	iterables()	Returns a PTransform that takes a PCollection<Iterable<T>> and returns a PCollection<T> containing all the elements from all the Iterables.
static <T> Flatten.PCollections<T>	pCollections()	Returns a PTransform that flattens a PCollectionList into a PCollection containing all the elements of all the PCollections in its input.
static <T> PTransform<PCollection<T>,PCollection<T>>	with(PTransform<PBegin,PCollection<T>> other)	Returns a PTransform that flattens the input PCollection with the output of another PTransform resulting in a PCollection containing all the elements of both the input PCollections and the output of the given PTransform as its output.
static <T> PTransform<PCollection<T>,PCollection<T>>	with(PCollection<T> other)	Returns a PTransform that flattens the input PCollection with a given PCollection resulting in a PCollection containing all the elements of both PCollections as its output.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Flatten

public Flatten()

Method Details

pCollections

public static <T> Flatten.PCollections<T> pCollections()

Returns a PTransform that flattens a PCollectionList into a PCollection containing all the elements of all the PCollections in its input.

All inputs must have equal WindowFns. The output elements of Flatten<T> are in the same windows and have the same timestamps as their corresponding input elements. The output PCollection will have the same WindowFn as all of the inputs.

Type Parameters:

T - the type of the elements in the input and output PCollections.

iterables

public static <T> Flatten.Iterables<T> iterables()

Returns a PTransform that takes a PCollection<Iterable<T>> and returns a PCollection<T> containing all the elements from all the Iterables.

Example of use:

 PCollection<Iterable<Integer>> pcOfIterables = ...;
 PCollection<Integer> pc = pcOfIterables.apply(Flatten.<Integer>iterables());
 

By default, the output PCollection encodes its elements using the same Coder that the input uses for the elements in its Iterable.

Type Parameters:

T - the type of the elements of the input Iterable and the output PCollection

with

public static <T>
PTransform<PCollection<T>,PCollection<T>> with(PCollection<T> other)

Returns a PTransform that flattens the input PCollection with a given PCollection resulting in a PCollection containing all the elements of both PCollections as its output.

This is equivalent to creating a PCollectionList containing both the input and other and then applying pCollections(), but has the advantage that it can be more easily used inline.

Both must have equal WindowFns. The output elements of Flatten<T> are in the same windows and have the same timestamps as their corresponding input elements. The output PCollection will have the same WindowFn as both inputs.

Type Parameters:

T - the type of the elements in the input and output PCollections.

Parameters:

other - the other PCollection to flatten with the input

with

public static <T>
PTransform<PCollection<T>,PCollection<T>> with(PTransform<PBegin,PCollection<T>> other)

Returns a PTransform that flattens the input PCollection with the output of another PTransform resulting in a PCollection containing all the elements of both the input PCollections and the output of the given PTransform as its output.

This is equivalent to creating a PCollectionList containing both the input and the output of other and then applying pCollections(), but has the advantage that it can be more easily used inline.

Both PCollections must have equal WindowFns. The output elements of Flatten<T> are in the same windows and have the same timestamps as their corresponding input elements. The output PCollection will have the same WindowFn as both inputs.

Type Parameters:

T - the type of the elements in the input and output PCollections.

Parameters:

other - a PTransform whose ouptput should be flattened with the input