apache_beam.runners.interactive.interactive_runner module

A runner that allows running of Beam pipelines interactively.

This module is experimental. No backwards-compatibility guarantees.

class apache_beam.runners.interactive.interactive_runner.InteractiveRunner(underlying_runner=None, cache_dir=None, cache_format='text', render_option=None, skip_display=True, force_compute=True, blocking=True)

Bases: apache_beam.runners.runner.PipelineRunner

An interactive runner for Beam Python pipelines.

Allows interactively building and running Beam Python pipelines.

Constructor of InteractiveRunner.

Parameters:

• underlying_runner – (runner.PipelineRunner)
• cache_dir – (str) the directory where PCollection caches are kept
• cache_format – (str) the file format that should be used for saving PCollection caches. Available options are ‘text’ and ‘tfrecord’.
• render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.
• skip_display – (bool) whether to skip display operations when running the pipeline. Useful if running large pipelines when display is not needed.
• force_compute – (bool) whether sequential pipeline runs can use cached data of PCollections computed from the previous runs including show API invocation from interactive_beam module. If True, always run the whole pipeline and compute data for PCollections forcefully. If False, use available data and run minimum pipeline fragment to only compute data not available.
• blocking – (bool) whether the pipeline run should be blocking or not.

is_fnapi_compatible()

set_render_option(render_option)

Sets the rendering option.

Parameters:render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.

start_session()

Start the session that keeps back-end managers and workers alive.

end_session()

End the session that keeps backend managers and workers alive.

cleanup()

apply(transform, pvalueish, options)

run_pipeline(pipeline, options)

class apache_beam.runners.interactive.interactive_runner.PipelineResult(underlying_result, pipeline_instrument)

Bases: apache_beam.runners.runner.PipelineResult

Provides access to information about a pipeline.

Constructor of PipelineResult.

Parameters:

• underlying_result – (PipelineResult) the result returned by the underlying runner running the pipeline.
• pipeline_instrument – (PipelineInstrument) pipeline instrument describing the pipeline being executed with interactivity applied and related metadata including where the interactivity-backing cache lies.

state

wait_until_finish()

get(pcoll, include_window_info=False)

Materializes the PCollection into a list.

If include_window_info is True, then returns the elements as WindowedValues. Otherwise, return the element as itself.

read(pcoll, include_window_info=False)

Reads the PCollection one element at a time from cache.

If include_window_info is True, then returns the elements as WindowedValues. Otherwise, return the element as itself.

cancel()