apache_beam.runners.interactive.interactive_runner module

A runner that allows running of Beam pipelines interactively.

This module is experimental. No backwards-compatibility guarantees.

class apache_beam.runners.interactive.interactive_runner.InteractiveRunner(underlying_runner=None, cache_dir=None, cache_format='text', render_option=None, skip_display=False)[source]

Bases: apache_beam.runners.runner.PipelineRunner

An interactive runner for Beam Python pipelines.

Allows interactively building and running Beam Python pipelines.

Constructor of InteractiveRunner.

Parameters:
  • underlying_runner – (runner.PipelineRunner)
  • cache_dir – (str) the directory where PCollection caches are kept
  • cache_format – (str) the file format that should be used for saving PCollection caches. Available options are ‘text’ and ‘tfrecord’.
  • render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.
  • skip_display – (bool) whether to skip display operations when running the pipeline. Useful if running large pipelines when display is not needed.
is_fnapi_compatible()[source]
set_render_option(render_option)[source]

Sets the rendering option.

Parameters:render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.
start_session()[source]

Start the session that keeps back-end managers and workers alive.

end_session()[source]

End the session that keeps backend managers and workers alive.

cleanup()[source]
apply(transform, pvalueish, options)[source]
run_pipeline(pipeline, options)[source]
class apache_beam.runners.interactive.interactive_runner.PipelineResult(underlying_result, pipeline_instrument)[source]

Bases: apache_beam.runners.runner.PipelineResult

Provides access to information about a pipeline.

Constructor of PipelineResult.

Parameters:
  • underlying_result – (PipelineResult) the result returned by the underlying runner running the pipeline.
  • pipeline_instrument – (PipelineInstrument) pipeline instrument describing the pipeline being executed with interactivity applied and related metadata including where the interactivity-backing cache lies.
wait_until_finish()[source]
get(pcoll)[source]
cancel()[source]