apache_beam.runners.interactive.interactive_runner module

A runner that allows running of Beam pipelines interactively.

This module is experimental. No backwards-compatibility guarantees.

class apache_beam.runners.interactive.interactive_runner.InteractiveRunner(underlying_runner=None, cache_dir=None, cache_format='text', render_option=None)

Bases: apache_beam.runners.runner.PipelineRunner

An interactive runner for Beam Python pipelines.

Allows interactively building and running Beam Python pipelines.

Constructor of InteractiveRunner.

Parameters:

• underlying_runner – (runner.PipelineRunner)
• cache_dir – (str) the directory where PCollection caches are kept
• cache_format – (str) the file format that should be used for saving PCollection caches. Available options are ‘text’ and ‘tfrecord’.
• render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.

is_fnapi_compatible()

set_render_option(render_option)

Sets the rendering option.

Parameters:render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.

start_session()

Start the session that keeps back-end managers and workers alive.

end_session()

End the session that keeps backend managers and workers alive.

cleanup()

apply(transform, pvalueish, options)

run_pipeline(pipeline, options)

class apache_beam.runners.interactive.interactive_runner.PipelineResult(underlying_result, runner, pipeline_info, cache_manager, pcolls_to_pcoll_id)

Bases: apache_beam.runners.runner.PipelineResult

Provides access to information about a pipeline.

wait_until_finish()

get(pcoll)

sample(pcoll)