public class Flatten
extends java.lang.Object
Flatten<T> takes multiple PCollection<T>s bundled into a PCollectionList<T> and returns a single PCollection<T> containing all the elements in
all the input PCollections. The name "Flatten" suggests taking a list of lists and
flattening them into a single list.
Example of use:
PCollection<String> pc1 = ...;
PCollection<String> pc2 = ...;
PCollection<String> pc3 = ...;
PCollectionList<String> pcs = PCollectionList.of(pc1).and(pc2).and(pc3);
PCollection<String> merged = pcs.apply(Flatten.<String>pCollections());
By default, the Coder of the output PCollection is the same as the Coder of the first PCollection in the input PCollectionList (if the PCollectionList is non-empty).
| Modifier and Type | Class and Description |
|---|---|
static class |
Flatten.Iterables<T>
FlattenIterables<T> takes a PCollection<Iterable<T>> and returns a PCollection<T> that contains all the elements from each iterable. |
static class |
Flatten.PCollections<T>
A
PTransform that flattens a PCollectionList into a PCollection
containing all the elements of all the PCollections in its input. |
| Constructor and Description |
|---|
Flatten() |
| Modifier and Type | Method and Description |
|---|---|
static <T> Flatten.Iterables<T> |
iterables()
Returns a
PTransform that takes a PCollection<Iterable<T>> and returns a PCollection<T> containing all the elements from all the Iterables. |
static <T> Flatten.PCollections<T> |
pCollections()
Returns a
PTransform that flattens a PCollectionList into a PCollection
containing all the elements of all the PCollections in its input. |
static <T> PTransform<PCollection<T>,PCollection<T>> |
with(PCollection<T> other)
Returns a
PTransform that flattens the input PCollection with a given PCollection resulting in a PCollection containing all the elements of both PCollections as its output. |
static <T> PTransform<PCollection<T>,PCollection<T>> |
with(PTransform<PBegin,PCollection<T>> other)
Returns a
PTransform that flattens the input PCollection with the output of
another PTransform resulting in a PCollection containing all the elements of
both the input PCollections and the output of the given PTransform as its
output. |
public static <T> Flatten.PCollections<T> pCollections()
PTransform that flattens a PCollectionList into a PCollection
containing all the elements of all the PCollections in its input.
All inputs must have equal WindowFns. The output elements of Flatten<T> are
in the same windows and have the same timestamps as their corresponding input elements. The
output PCollection will have the same WindowFn as all of the inputs.
T - the type of the elements in the input and output PCollections.public static <T> Flatten.Iterables<T> iterables()
PTransform that takes a PCollection<Iterable<T>> and returns a PCollection<T> containing all the elements from all the Iterables.
Example of use:
PCollection<Iterable<Integer>> pcOfIterables = ...;
PCollection<Integer> pc = pcOfIterables.apply(Flatten.<Integer>iterables());
By default, the output PCollection encodes its elements using the same Coder
that the input uses for the elements in its Iterable.
T - the type of the elements of the input Iterable and the output PCollectionpublic static <T> PTransform<PCollection<T>,PCollection<T>> with(PCollection<T> other)
PTransform that flattens the input PCollection with a given PCollection resulting in a PCollection containing all the elements of both PCollections as its output.
This is equivalent to creating a PCollectionList containing both the input and
other and then applying pCollections(), but has the advantage that it can be
more easily used inline.
Both must have equal WindowFns. The output elements of Flatten<T> are in the same windows and have the same timestamps as their corresponding input
elements. The output PCollection will have the same WindowFn as both inputs.
T - the type of the elements in the input and output PCollections.other - the other PCollection to flatten with the inputpublic static <T> PTransform<PCollection<T>,PCollection<T>> with(PTransform<PBegin,PCollection<T>> other)
PTransform that flattens the input PCollection with the output of
another PTransform resulting in a PCollection containing all the elements of
both the input PCollections and the output of the given PTransform as its
output.
This is equivalent to creating a PCollectionList containing both the input and the
output of other and then applying pCollections(), but has the advantage that
it can be more easily used inline.
Both PCollections must have equal WindowFns. The output elements of Flatten<T> are in the same windows and have the same timestamps as their corresponding input
elements. The output PCollection will have the same WindowFn as both inputs.
T - the type of the elements in the input and output PCollections.other - a PTransform whose ouptput should be flattened with the input