Class CompositePipeline

java.lang.Object
org.apache.beam.sdk.Pipeline
org.apache.beam.sdk.CompositePipeline

@Internal public final class CompositePipeline extends Pipeline
For internal use only; no backwards-compatibility guarantees.

A read-only Pipeline that presents several independently constructed member pipelines as one graph, so that a runner can translate and execute them together as a single job.

Only the two operations runners use to translate a pipeline are composed: traverseTopologically(org.apache.beam.sdk.Pipeline.PipelineVisitor) visits every member in turn, and replaceAll(java.util.List<org.apache.beam.sdk.runners.PTransformOverride>) applies the runner's overrides to every member. Everything else is the composite's own: Pipeline.getOptions() are the options passed to the constructor (member options are ignored), the coder and schema registries are fresh and empty, and transforms cannot be applied to it.

Visitor contract: traverseTopologically(org.apache.beam.sdk.Pipeline.PipelineVisitor) invokes Pipeline.PipelineVisitor.enterPipeline(org.apache.beam.sdk.Pipeline) and Pipeline.PipelineVisitor.leavePipeline(org.apache.beam.sdk.Pipeline) once per member, each time with that member, and visits the member's root node in between; it never passes the composite itself to the visitor. Consequently Pipeline.PipelineVisitor.Defaults.getPipeline() always yields the member that owns the node being visited, which is what visitors needing a member's state (for example its Pipeline.getCoderRegistry()) should use. Visitors that assume a single enterPipeline/root-node visit per traversal must instead be applied to each member in getMemberPipelines().

Members must have pairwise distinct top-level transform names (and therefore distinct full names throughout, since every nested name and PCollection name is derived from its enclosing transform's); this is checked at construction. The simplest way to guarantee it is to give each member a unique root name via TransformHierarchy.setRootName(java.lang.String) before applying any transform.