apache_beam.runners.interactive.interactive_runner module

A runner that allows running of Beam pipelines interactively.

This module is experimental. No backwards-compatibility guarantees.

class apache_beam.runners.interactive.interactive_runner.InteractiveRunner(underlying_runner=None, cache_dir=None, cache_format='text', render_option=None, skip_display=False)

Bases: apache_beam.runners.runner.PipelineRunner

An interactive runner for Beam Python pipelines.

Allows interactively building and running Beam Python pipelines.

Constructor of InteractiveRunner.

Parameters:

• underlying_runner – (runner.PipelineRunner)
• cache_dir – (str) the directory where PCollection caches are kept
• cache_format – (str) the file format that should be used for saving PCollection caches. Available options are ‘text’ and ‘tfrecord’.
• render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.
• skip_display – (bool) whether to skip display operations when running the pipeline. Useful if running large pipelines when display is not needed.

is_fnapi_compatible()

set_render_option(render_option)

Sets the rendering option.

Parameters:render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.

start_session()

Start the session that keeps back-end managers and workers alive.

end_session()

End the session that keeps backend managers and workers alive.

cleanup()

apply(transform, pvalueish, options)

run_pipeline(pipeline, options)

class apache_beam.runners.interactive.interactive_runner.PipelineResult(underlying_result, pipeline_instrument)

Bases: apache_beam.runners.runner.PipelineResult

Provides access to information about a pipeline.

Constructor of PipelineResult.

Parameters:

• underlying_result – (PipelineResult) the result returned by the underlying runner running the pipeline.
• pipeline_instrument – (PipelineInstrument) pipeline instrument describing the pipeline being executed with interactivity applied and related metadata including where the interactivity-backing cache lies.

wait_until_finish()

get(pcoll)

cancel()