apache_beam.runners.interactive.pipeline_fragment module

Module to build pipeline fragment that produces given PCollections.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.runners.interactive.pipeline_fragment.PipelineFragment(pcolls, options=None)[source]

Bases: object

A fragment of a pipeline definition.

A pipeline fragment is built from the original pipeline definition to include only PTransforms that are necessary to produce the given PCollections.

Constructor of PipelineFragment.

Parameters:
  • pcolls – (List[PCollection]) a list of PCollections to build pipeline fragment for.
  • options – (PipelineOptions) the pipeline options for the implicit pipeline run.
deduce_fragment()[source]

Deduce the pipeline fragment as an apache_beam.Pipeline instance.

run(display_pipeline_graph=False, use_cache=True, blocking=False)[source]

Shorthand to run the pipeline fragment.

apache_beam.runners.interactive.pipeline_fragment.should_skip_pruning(transform: apache_beam.pipeline.AppliedPTransform)[source]