Class PCollectionTuple
PCollectionTuple
is an immutable tuple of heterogeneously-typed PCollections
, "keyed" by TupleTags
. A PCollectionTuple
can be used as
the input or output of a PTransform
taking or producing multiple PCollection inputs or
outputs that can be of different types, for instance a ParDo
with multiple outputs.
A PCollectionTuple
can be created and accessed like follows:
PCollection<String> pc1 = ...;
PCollection<Integer> pc2 = ...;
PCollection<Iterable<String>> pc3 = ...;
// Create TupleTags for each of the PCollections to put in the
// PCollectionTuple (the type of the TupleTag enables tracking the
// static type of each of the PCollections in the PCollectionTuple):
TupleTag<String> tag1 = new TupleTag<>();
TupleTag<Integer> tag2 = new TupleTag<>();
TupleTag<Iterable<String>> tag3 = new TupleTag<>();
// Create a PCollectionTuple with three PCollections:
PCollectionTuple pcs =
PCollectionTuple.of(tag1, pc1)
.and(tag2, pc2)
.and(tag3, pc3);
// Create an empty PCollectionTuple:
Pipeline p = ...;
PCollectionTuple pcs2 = PCollectionTuple.empty(p);
// Get PCollections out of a PCollectionTuple, using the same tags
// that were used to put them in:
PCollection<Integer> pcX = pcs.get(tag2);
PCollection<String> pcY = pcs.get(tag1);
PCollection<Iterable<String>> pcZ = pcs.get(tag3);
// Get a map of all PCollections in a PCollectionTuple:
Map<TupleTag<?>, PCollection<?>> allPcs = pcs.getAll();
-
Method Summary
Modifier and TypeMethodDescription<T> PCollectionTuple
and
(String tag, PCollection<T> pc) A version ofand(TupleTag, PCollection)
that takes in a String instead of a TupleTag.<T> PCollectionTuple
and
(TupleTag<T> tag, PCollection<T> pc) Returns a newPCollectionTuple
that has eachPCollection
andTupleTag
of thisPCollectionTuple
plus the givenPCollection
associated with the givenTupleTag
.<OutputT extends POutput>
OutputTapply
(String name, PTransform<? super PCollectionTuple, OutputT> t) Applies the givenPTransform
to this inputPCollectionTuple
, usingname
to identify this specific application of the transform.<OutputT extends POutput>
OutputTapply
(PTransform<? super PCollectionTuple, OutputT> t) Likeapply(String, PTransform)
but defaulting to the name of thePTransform
.static PCollectionTuple
Returns an emptyPCollectionTuple
that is part of the givenPipeline
.boolean
expand()
void
finishSpecifyingOutput
(String transformName, PInput input, PTransform<?, ?> transform) As part of applying the producingPTransform
, finalizes this output to make it ready for being used as an input and for running.<T> PCollection
<T> Returns thePCollection
associated with the given tag in thisPCollectionTuple
.<T> PCollection
<T> Map
<TupleTag<?>, PCollection<?>> getAll()
Returns an immutable Map fromTupleTag
to correspondingPCollection
, for all the members of thisPCollectionTuple
.<T> boolean
Returns whether thisPCollectionTuple
contains aPCollection
with the given tag.<T> boolean
Returns whether thisPCollectionTuple
contains aPCollection
with the given tag.int
hashCode()
static <T> PCollectionTuple
of
(String tag, PCollection<T> pc) A version ofof(TupleTag, PCollection)
that takes in a String instead of aTupleTag
.static <T> PCollectionTuple
of
(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2) A version ofof(String, PCollection)
that takes in two PCollections of the same type.static <T> PCollectionTuple
of
(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3) A version ofof(String, PCollection)
that takes in three PCollections of the same type.static <T> PCollectionTuple
of
(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3, String tag4, PCollection<T> pc4) A version ofof(String, PCollection)
that takes in four PCollections of the same type.static <T> PCollectionTuple
of
(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3, String tag4, PCollection<T> pc4, String tag5, PCollection<T> pc5) A version ofof(String, PCollection)
that takes in five PCollections of the same type.static <T> PCollectionTuple
of
(TupleTag<T> tag, PCollection<T> pc) static PCollectionTuple
ofPrimitiveOutputsInternal
(Pipeline pipeline, TupleTagList outputTags, Map<TupleTag<?>, Coder<?>> coders, WindowingStrategy<?, ?> windowingStrategy, PCollection.IsBounded isBounded) For internal use only; no backwards-compatibility guarantees.
-
Method Details
-
empty
Returns an emptyPCollectionTuple
that is part of the givenPipeline
.A
PCollectionTuple
containing additional elements can be created by callingand(org.apache.beam.sdk.values.TupleTag<T>, org.apache.beam.sdk.values.PCollection<T>)
on the result. -
of
Returns a singletonPCollectionTuple
containing the givenPCollection
keyed by the givenTupleTag
.A
PCollectionTuple
containing additional elements can be created by callingand(org.apache.beam.sdk.values.TupleTag<T>, org.apache.beam.sdk.values.PCollection<T>)
on the result. -
of
A version ofof(TupleTag, PCollection)
that takes in a String instead of aTupleTag
.This method is simpler for cases when a typed tuple-tag is not needed to extract a PCollection, for example when using schema transforms.
-
of
public static <T> PCollectionTuple of(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2) A version ofof(String, PCollection)
that takes in two PCollections of the same type. -
of
public static <T> PCollectionTuple of(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3) A version ofof(String, PCollection)
that takes in three PCollections of the same type. -
of
public static <T> PCollectionTuple of(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3, String tag4, PCollection<T> pc4) A version ofof(String, PCollection)
that takes in four PCollections of the same type. -
of
public static <T> PCollectionTuple of(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3, String tag4, PCollection<T> pc4, String tag5, PCollection<T> pc5) A version ofof(String, PCollection)
that takes in five PCollections of the same type. -
and
Returns a newPCollectionTuple
that has eachPCollection
andTupleTag
of thisPCollectionTuple
plus the givenPCollection
associated with the givenTupleTag
.The given
TupleTag
should not already be mapped to aPCollection
in thisPCollectionTuple
.Each
PCollection
in the resultingPCollectionTuple
must be part of the samePipeline
. -
and
A version ofand(TupleTag, PCollection)
that takes in a String instead of a TupleTag.This method is simpler for cases when a typed tuple-tag is not needed to extract a PCollection, for example when using schema transforms.
-
has
Returns whether thisPCollectionTuple
contains aPCollection
with the given tag. -
has
Returns whether thisPCollectionTuple
contains aPCollection
with the given tag. -
get
Returns thePCollection
associated with the givenTupleTag
in thisPCollectionTuple
. ThrowsIllegalArgumentException
if there is no suchPCollection
, i.e.,!has(tag)
. -
get
Returns thePCollection
associated with the given tag in thisPCollectionTuple
. ThrowsIllegalArgumentException
if there is no suchPCollection
, i.e.,!has(tag)
. -
getAll
Returns an immutable Map fromTupleTag
to correspondingPCollection
, for all the members of thisPCollectionTuple
. -
apply
Likeapply(String, PTransform)
but defaulting to the name of thePTransform
.- Returns:
- the output of the applied
PTransform
-
apply
public <OutputT extends POutput> OutputT apply(String name, PTransform<? super PCollectionTuple, OutputT> t) Applies the givenPTransform
to this inputPCollectionTuple
, usingname
to identify this specific application of the transform. This name is used in various places, including the monitoring UI, logging, and to stably identify this application node in the job graph.- Returns:
- the output of the applied
PTransform
-
ofPrimitiveOutputsInternal
@Internal public static PCollectionTuple ofPrimitiveOutputsInternal(Pipeline pipeline, TupleTagList outputTags, Map<TupleTag<?>, Coder<?>> coders, WindowingStrategy<?, ?> windowingStrategy, PCollection.IsBounded isBounded) For internal use only; no backwards-compatibility guarantees.Returns a
PCollectionTuple
with each of the given tags mapping to a new outputPCollection
.For use by primitive transformations only.
-
getPipeline
Description copied from interface:PInput
- Specified by:
getPipeline
in interfacePInput
- Specified by:
getPipeline
in interfacePOutput
-
expand
Description copied from interface:PInput
Expands thisPInput
into a list of its component outputPValues
.- A
PValue
expands to itself. - A tuple or list of
PValues
(such asPCollectionTuple
orPCollectionList
) expands to its componentPValue PValues
.
Not intended to be invoked directly by user code.
- A
-
finishSpecifyingOutput
Description copied from interface:POutput
As part of applying the producingPTransform
, finalizes this output to make it ready for being used as an input and for running.This includes ensuring that all
PCollections
haveCoders
specified or defaulted.Automatically invoked whenever this
POutput
is output, afterPOutput.finishSpecifyingOutput(String, PInput, PTransform)
has been called on each componentPValue
returned byPOutput.expand()
.- Specified by:
finishSpecifyingOutput
in interfacePOutput
-
equals
-
hashCode
public int hashCode()
-