apache_beam.runners.interactive.pipeline_fragment module

Module to build pipeline fragment that produces given PCollections.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.runners.interactive.pipeline_fragment.PipelineFragment(pcolls, options=None, runner=None)[source]

Bases: object

A fragment of a pipeline definition.

A pipeline fragment is built from the original pipeline definition to include only PTransforms that are necessary to produce the given PCollections.

Constructor of PipelineFragment.

Parameters:

  • pcolls – (List[PCollection]) a list of PCollections to build pipeline fragment for.

  • options – (PipelineOptions) the pipeline options for the implicit pipeline run.

  • runner – (Runner) the pipeline runner for the implicit pipeline run.

deduce_fragment()[source]

Deduce the pipeline fragment as an apache_beam.Pipeline instance.

run(display_pipeline_graph=False, use_cache=True, blocking=False)[source]

Shorthand to run the pipeline fragment.

apache_beam.runners.interactive.pipeline_fragment.should_skip_pruning(transform: AppliedPTransform)[source]