apache_beam.runners.render module

A portable “runner” that renders a beam graph.

This runner can either render the graph to a (set of) output path(s), as designated by (possibly repeated) –render_output, or serve the pipeline as an interactive graph, if –render_port is set.

In Python, this runner can be passed directly at pipeline construction, e.g.:

with beam.Pipeline(runner=beam.runners.render.RenderRunner(), options=...)

For other languages, start this service a by running:

python -m apache_beam.runners.render --job_port=PORT ...

and then run your pipline with the PortableRunner setting the job endpoint to localhost:PORT.

If any –render_output=path.ext flags are passed, each submitted job will get written to the given output (overwriting any previously existing file).

If –render_port is set to a non-negative value, a local http server will be started which allows for interactive exploration of the pipeline graph.

As an alternative to starting a job server, a single pipeline can be rendered by passing a pipeline proto file to –pipeline_proto. For example

python -m apache_beam.runners.render

–pipeline_proto gs://<staging_location>/pipeline.pb –render_output=/tmp/pipeline.svg

Requires the graphviz dot executable to be available in the path.

class apache_beam.runners.render.RenderOptions(flags=None, **kwargs)

Bases: apache_beam.options.pipeline_options.PipelineOptions

Rendering options.

Initialize an options class.

The initializer will traverse all subclasses, add all their argparse arguments and then parse the command line specified by flags or by default the one obtained from sys.argv.

The subclasses of PipelineOptions do not need to redefine __init__.

Parameters:

• flags – An iterable of command line arguments to be used. If not specified then sys.argv will be used as input for parsing arguments.
• **kwargs – Add overrides for arguments passed in flags. For overrides of arguments, please pass the option names instead of flag names. Option names: These are defined as dest in the parser.add_argument() for each flag. Passing flags like {no_use_public_ips: True}, for which the dest is defined to a different flag name in the parser, would be discarded. Instead, pass the dest of the flag (dest of no_use_public_ips is use_public_ips).

class apache_beam.runners.render.PipelineRenderer(pipeline, options)

Bases: object

update(toggle=None)

style(transform_id)

to_dot()

to_dot_iter()

transform_to_dot(transform_id, pcoll_leaf_consumers, edges_out)

transform_node(transform_id)

transform_attributes(transform_id)

pcoll_leaf_consumers_iter(transform_id)

pcoll_leaf_consumers()

is_leaf(transform_id)

info()

layout_dot()

page_callback_data(layout)

render_data()

render_json()

page()

class apache_beam.runners.render.RenderRunner

Bases: apache_beam.runners.runner.PipelineRunner

run_pipeline(pipeline_object, options)

run_portable_pipeline(pipeline_proto, options)

class apache_beam.runners.render.RenderPipelineResult(server)

Bases: apache_beam.runners.runner.PipelineResult

wait_until_finish(duration=None)

monitoring_infos()

apache_beam.runners.render.run(argv)

apache_beam.runners.render.render_one(options)

apache_beam.runners.render.run_server(options)