apache_beam.runners.interactive.pipeline_analyzer module

Analyzes and modifies the pipeline that utilize the PCollection cache.

This module is experimental. No backwards-compatibility guarantees.

class apache_beam.runners.interactive.pipeline_analyzer.PipelineAnalyzer(cache_manager, pipeline_proto, underlying_runner, options=None, desired_cache_labels=None)

Bases: object

Constructor of PipelineAnanlyzer.

Parameters:

• cache_manager – (CacheManager)
• pipeline_proto – (Pipeline proto)
• underlying_runner – (PipelineRunner)
• options – (PipelineOptions)
• desired_cache_labels – (Set[str]) a set of labels of the PCollection queried by the user.

pipeline_info()

Return PipelineInfo of the original pipeline.

pipeline_proto_to_execute()

Returns Pipeline proto to be executed.

tl_referenced_pcoll_ids()

Returns a set of PCollection IDs referenced by top level PTransforms.

tl_required_trans_ids()

Returns a set of required top level PTransform IDs.

caches_used()

Returns a set of PCollection IDs to read from cache.

read_cache_ids()

Return a set of ReadCache PTransform IDs inserted.

write_cache_ids()

Return a set of WriteCache PTransform IDs inserted.

class apache_beam.runners.interactive.pipeline_analyzer.PipelineInfo(proto)

Bases: object

Provides access to pipeline metadata.

all_pcollections()

leaf_pcollections()

producer(pcoll_id)

cache_label(pcoll_id)

Returns the cache label given the PCollection ID.

class Derivation(inputs, transform_proto, output_tag)

Bases: object

Records derivation info of a PCollection. Helper for PipelineInfo.

Constructor of Derivation.

Parameters:

• inputs – (Dict[str, Derivation]) maps PCollection names to Derivations.
• transform_proto – (Transform proto) the producing PTransform.
• output_tag – (str) local name of the PCollection in analysis.

cache_label()

json()

apache_beam.runners.interactive.pipeline_analyzer.set_proto_map(proto_map, new_value)