Class CompositePipeline
A read-only Pipeline that presents several independently constructed member pipelines
as one graph, so that a runner can translate and execute them together as a single job.
Only the two operations runners use to translate a pipeline are composed: traverseTopologically(org.apache.beam.sdk.Pipeline.PipelineVisitor) visits every member in turn, and replaceAll(java.util.List<org.apache.beam.sdk.runners.PTransformOverride>) applies the runner's
overrides to every member. Everything else is the composite's own: Pipeline.getOptions() are the
options passed to the constructor (member options are ignored), the coder and schema registries
are fresh and empty, and transforms cannot be applied to it.
Visitor contract: traverseTopologically(org.apache.beam.sdk.Pipeline.PipelineVisitor) invokes Pipeline.PipelineVisitor.enterPipeline(org.apache.beam.sdk.Pipeline)
and Pipeline.PipelineVisitor.leavePipeline(org.apache.beam.sdk.Pipeline) once per member, each time with that member,
and visits the member's root node in between; it never passes the composite itself to the
visitor. Consequently Pipeline.PipelineVisitor.Defaults.getPipeline() always yields the member
that owns the node being visited, which is what visitors needing a member's state (for example
its Pipeline.getCoderRegistry()) should use. Visitors that assume a single
enterPipeline/root-node visit per traversal must instead be applied to each member in getMemberPipelines().
Members must have pairwise distinct top-level transform names (and therefore distinct full
names throughout, since every nested name and PCollection name
is derived from its enclosing transform's); this is checked at construction. The simplest way to
guarantee it is to give each member a unique root name via TransformHierarchy.setRootName(java.lang.String)
before applying any transform.
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.beam.sdk.Pipeline
Pipeline.PipelineExecutionException, Pipeline.PipelineVisitor -
Constructor Summary
ConstructorsConstructorDescriptionCompositePipeline(PipelineOptions options, List<Pipeline> memberPipelines) Creates a composite overmemberPipelines. -
Method Summary
Modifier and TypeMethodDescription<OutputT extends POutput>
OutputTapply(String name, PTransform<? super PBegin, OutputT> root) <OutputT extends POutput>
OutputTapply(PTransform<? super PBegin, OutputT> root) LikePipeline.apply(String, PTransform)but the transform node in thePipelinegraph will be named according toPTransform.getName().The member pipelines, in the order they were given and are traversed.voidreplaceAll(List<org.apache.beam.sdk.runners.PTransformOverride> overrides) For internal use only; no backwards-compatibility guarantees.voidFor internal use only; no backwards-compatibility guarantees.Methods inherited from class org.apache.beam.sdk.Pipeline
applyTransform, applyTransform, begin, create, create, forTransformHierarchy, getCoderRegistry, getOptions, getSchemaRegistry, registerBadRecordErrorHandler, run, run, setCoderRegistry, toString
-
Constructor Details
-
CompositePipeline
Creates a composite overmemberPipelines.- Throws:
IllegalArgumentException- if two members share a top-level transform name
-
-
Method Details
-
getMemberPipelines
The member pipelines, in the order they were given and are traversed. -
traverseTopologically
Description copied from class:PipelineFor internal use only; no backwards-compatibility guarantees.Invokes the
PipelineVisitor'sPipeline.PipelineVisitor.visitPrimitiveTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node)andPipeline.PipelineVisitor.visitValue(org.apache.beam.sdk.values.PValue, org.apache.beam.sdk.runners.TransformHierarchy.Node)operations on each of thisPipeline'stransform and value nodes, in forward topological order.Traversal of the
PipelinecausesPTransformsandPValuesowned by thePipelineto be marked as finished, at which point they may no longer be modified.Typically invoked by
PipelineRunnersubclasses.- Overrides:
traverseTopologicallyin classPipeline
-
replaceAll
Description copied from class:PipelineFor internal use only; no backwards-compatibility guarantees.Replaces all nodes that match a
PTransformOverridein this pipeline. Overrides are applied in the order they are present within the list.- Overrides:
replaceAllin classPipeline
-
apply
Description copied from class:PipelineLikePipeline.apply(String, PTransform)but the transform node in thePipelinegraph will be named according toPTransform.getName().- Overrides:
applyin classPipeline- Throws:
UnsupportedOperationException- always; a composite is a read-only view.- See Also:
-
apply
public <OutputT extends POutput> OutputT apply(String name, PTransform<? super PBegin, OutputT> root) Description copied from class:PipelineAdds a rootPTransform, such asReadorCreate, to thisPipeline.The node in the
Pipelinegraph will use the providedname. This name is used in various places, including the monitoring UI, logging, and to stably identify this node in thePipelinegraph upon update.Alias for
begin().apply(name, root).- Overrides:
applyin classPipeline- Throws:
UnsupportedOperationException- always; a composite is a read-only view.
-