apache_beam.runners.interactive.interactive_runner module

A runner that allows running of Beam pipelines interactively.

This module is experimental. No backwards-compatibility guarantees.

class apache_beam.runners.interactive.interactive_runner.InteractiveRunner(underlying_runner=None, cache_dir=None, cache_format='text', render_option=None)[source]

Bases: apache_beam.runners.runner.PipelineRunner

An interactive runner for Beam Python pipelines.

Allows interactively building and running Beam Python pipelines.

Constructor of InteractiveRunner.

Parameters:
  • underlying_runner – (runner.PipelineRunner)
  • cache_dir – (str) the directory where PCollection caches are kept
  • cache_format – (str) the file format that should be used for saving PCollection caches. Available options are ‘text’ and ‘tfrecord’.
  • render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.
is_fnapi_compatible()[source]
set_render_option(render_option)[source]

Sets the rendering option.

Parameters:render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.
start_session()[source]

Start the session that keeps back-end managers and workers alive.

end_session()[source]

End the session that keeps backend managers and workers alive.

cleanup()[source]
apply(transform, pvalueish, options)[source]
run_pipeline(pipeline, options)[source]
class apache_beam.runners.interactive.interactive_runner.PipelineResult(underlying_result, runner, pipeline_info, cache_manager, pcolls_to_pcoll_id)[source]

Bases: apache_beam.runners.runner.PipelineResult

Provides access to information about a pipeline.

wait_until_finish()[source]
get(pcoll)[source]
sample(pcoll)[source]