Class PCollectionTuple
PCollectionTuple is an immutable tuple of heterogeneously-typed PCollections, "keyed" by TupleTags. A PCollectionTuple can be used as
the input or output of a PTransform taking or producing multiple PCollection inputs or
outputs that can be of different types, for instance a ParDo with multiple outputs.
A PCollectionTuple can be created and accessed like follows:
PCollection<String> pc1 = ...;
PCollection<Integer> pc2 = ...;
PCollection<Iterable<String>> pc3 = ...;
// Create TupleTags for each of the PCollections to put in the
// PCollectionTuple (the type of the TupleTag enables tracking the
// static type of each of the PCollections in the PCollectionTuple):
TupleTag<String> tag1 = new TupleTag<>();
TupleTag<Integer> tag2 = new TupleTag<>();
TupleTag<Iterable<String>> tag3 = new TupleTag<>();
// Create a PCollectionTuple with three PCollections:
PCollectionTuple pcs =
PCollectionTuple.of(tag1, pc1)
.and(tag2, pc2)
.and(tag3, pc3);
// Create an empty PCollectionTuple:
Pipeline p = ...;
PCollectionTuple pcs2 = PCollectionTuple.empty(p);
// Get PCollections out of a PCollectionTuple, using the same tags
// that were used to put them in:
PCollection<Integer> pcX = pcs.get(tag2);
PCollection<String> pcY = pcs.get(tag1);
PCollection<Iterable<String>> pcZ = pcs.get(tag3);
// Get a map of all PCollections in a PCollectionTuple:
Map<TupleTag<?>, PCollection<?>> allPcs = pcs.getAll();
-
Method Summary
Modifier and TypeMethodDescription<T> PCollectionTupleand(String tag, PCollection<T> pc) A version ofand(TupleTag, PCollection)that takes in a String instead of a TupleTag.<T> PCollectionTupleand(TupleTag<T> tag, PCollection<T> pc) Returns a newPCollectionTuplethat has eachPCollectionandTupleTagof thisPCollectionTupleplus the givenPCollectionassociated with the givenTupleTag.<OutputT extends POutput>
OutputTapply(String name, PTransform<? super PCollectionTuple, OutputT> t) Applies the givenPTransformto this inputPCollectionTuple, usingnameto identify this specific application of the transform.<OutputT extends POutput>
OutputTapply(PTransform<? super PCollectionTuple, OutputT> t) Likeapply(String, PTransform)but defaulting to the name of thePTransform.static PCollectionTupleReturns an emptyPCollectionTuplethat is part of the givenPipeline.booleanexpand()voidfinishSpecifyingOutput(String transformName, PInput input, PTransform<?, ?> transform) As part of applying the producingPTransform, finalizes this output to make it ready for being used as an input and for running.<T> PCollection<T> Returns thePCollectionassociated with the given tag in thisPCollectionTuple.<T> PCollection<T> Map<TupleTag<?>, PCollection<?>> getAll()Returns an immutable Map fromTupleTagto correspondingPCollection, for all the members of thisPCollectionTuple.<T> booleanReturns whether thisPCollectionTuplecontains aPCollectionwith the given tag.<T> booleanReturns whether thisPCollectionTuplecontains aPCollectionwith the given tag.inthashCode()static <T> PCollectionTupleof(String tag, PCollection<T> pc) A version ofof(TupleTag, PCollection)that takes in a String instead of aTupleTag.static <T> PCollectionTupleof(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2) A version ofof(String, PCollection)that takes in two PCollections of the same type.static <T> PCollectionTupleof(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3) A version ofof(String, PCollection)that takes in three PCollections of the same type.static <T> PCollectionTupleof(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3, String tag4, PCollection<T> pc4) A version ofof(String, PCollection)that takes in four PCollections of the same type.static <T> PCollectionTupleof(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3, String tag4, PCollection<T> pc4, String tag5, PCollection<T> pc5) A version ofof(String, PCollection)that takes in five PCollections of the same type.static <T> PCollectionTupleof(TupleTag<T> tag, PCollection<T> pc) static PCollectionTupleofPrimitiveOutputsInternal(Pipeline pipeline, TupleTagList outputTags, Map<TupleTag<?>, Coder<?>> coders, WindowingStrategy<?, ?> windowingStrategy, PCollection.IsBounded isBounded) For internal use only; no backwards-compatibility guarantees.
-
Method Details
-
empty
Returns an emptyPCollectionTuplethat is part of the givenPipeline.A
PCollectionTuplecontaining additional elements can be created by callingand(org.apache.beam.sdk.values.TupleTag<T>, org.apache.beam.sdk.values.PCollection<T>)on the result. -
of
Returns a singletonPCollectionTuplecontaining the givenPCollectionkeyed by the givenTupleTag.A
PCollectionTuplecontaining additional elements can be created by callingand(org.apache.beam.sdk.values.TupleTag<T>, org.apache.beam.sdk.values.PCollection<T>)on the result. -
of
A version ofof(TupleTag, PCollection)that takes in a String instead of aTupleTag.This method is simpler for cases when a typed tuple-tag is not needed to extract a PCollection, for example when using schema transforms.
-
of
public static <T> PCollectionTuple of(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2) A version ofof(String, PCollection)that takes in two PCollections of the same type. -
of
public static <T> PCollectionTuple of(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3) A version ofof(String, PCollection)that takes in three PCollections of the same type. -
of
public static <T> PCollectionTuple of(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3, String tag4, PCollection<T> pc4) A version ofof(String, PCollection)that takes in four PCollections of the same type. -
of
public static <T> PCollectionTuple of(String tag1, PCollection<T> pc1, String tag2, PCollection<T> pc2, String tag3, PCollection<T> pc3, String tag4, PCollection<T> pc4, String tag5, PCollection<T> pc5) A version ofof(String, PCollection)that takes in five PCollections of the same type. -
and
Returns a newPCollectionTuplethat has eachPCollectionandTupleTagof thisPCollectionTupleplus the givenPCollectionassociated with the givenTupleTag.The given
TupleTagshould not already be mapped to aPCollectionin thisPCollectionTuple.Each
PCollectionin the resultingPCollectionTuplemust be part of the samePipeline. -
and
A version ofand(TupleTag, PCollection)that takes in a String instead of a TupleTag.This method is simpler for cases when a typed tuple-tag is not needed to extract a PCollection, for example when using schema transforms.
-
has
Returns whether thisPCollectionTuplecontains aPCollectionwith the given tag. -
has
Returns whether thisPCollectionTuplecontains aPCollectionwith the given tag. -
get
Returns thePCollectionassociated with the givenTupleTagin thisPCollectionTuple. ThrowsIllegalArgumentExceptionif there is no suchPCollection, i.e.,!has(tag). -
get
Returns thePCollectionassociated with the given tag in thisPCollectionTuple. ThrowsIllegalArgumentExceptionif there is no suchPCollection, i.e.,!has(tag). -
getAll
Returns an immutable Map fromTupleTagto correspondingPCollection, for all the members of thisPCollectionTuple. -
apply
Likeapply(String, PTransform)but defaulting to the name of thePTransform.- Returns:
- the output of the applied
PTransform
-
apply
public <OutputT extends POutput> OutputT apply(String name, PTransform<? super PCollectionTuple, OutputT> t) Applies the givenPTransformto this inputPCollectionTuple, usingnameto identify this specific application of the transform. This name is used in various places, including the monitoring UI, logging, and to stably identify this application node in the job graph.- Returns:
- the output of the applied
PTransform
-
ofPrimitiveOutputsInternal
@Internal public static PCollectionTuple ofPrimitiveOutputsInternal(Pipeline pipeline, TupleTagList outputTags, Map<TupleTag<?>, Coder<?>> coders, WindowingStrategy<?, ?> windowingStrategy, PCollection.IsBounded isBounded) For internal use only; no backwards-compatibility guarantees.Returns a
PCollectionTuplewith each of the given tags mapping to a new outputPCollection.For use by primitive transformations only.
-
getPipeline
Description copied from interface:PInput- Specified by:
getPipelinein interfacePInput- Specified by:
getPipelinein interfacePOutput
-
expand
Description copied from interface:PInputExpands thisPInputinto a list of its component outputPValues.- A
PValueexpands to itself. - A tuple or list of
PValues(such asPCollectionTupleorPCollectionList) expands to its componentPValue PValues.
Not intended to be invoked directly by user code.
- A
-
finishSpecifyingOutput
Description copied from interface:POutputAs part of applying the producingPTransform, finalizes this output to make it ready for being used as an input and for running.This includes ensuring that all
PCollectionshaveCodersspecified or defaulted.Automatically invoked whenever this
POutputis output, afterPOutput.finishSpecifyingOutput(String, PInput, PTransform)has been called on each componentPValuereturned byPOutput.expand().- Specified by:
finishSpecifyingOutputin interfacePOutput
-
equals
-
hashCode
public int hashCode()
-