Module to build pipeline fragment that produces given PCollections.
For internal use only; no backwards-compatibility guarantees.
A fragment of a pipeline definition.
A pipeline fragment is built from the original pipeline definition to include only PTransforms that are necessary to produce the given PCollections.
Constructor of PipelineFragment.
- pcolls – (List[PCollection]) a list of PCollections to build pipeline fragment for.
- options – (PipelineOptions) the pipeline options for the implicit pipeline run.
Deduce the pipeline fragment as an apache_beam.Pipeline instance.
run(display_pipeline_graph=False, use_cache=True, blocking=False)¶
Shorthand to run the pipeline fragment.