apache_beam.runners.interactive.messaging.interactive_environment_inspector module

Messaging mechanism to inspect the interactive environment.

A singleton instance is accessible from interactive_environment.current_env().inspector.

class apache_beam.runners.interactive.messaging.interactive_environment_inspector.InteractiveEnvironmentInspector(ignore_synthetic=True)

Bases: object

Inspector that converts information of the current interactive environment including pipelines and pcollections into JSON data suitable for messaging with applications within/outside the Python kernel.

The usage is always that the application side reads the inspectables or list_inspectables first then communicates back to the kernel and get_val for usage on the kernel side.

inspectables

Lists pipelines and pcollections assigned to variables as inspectables.

inspectable_pipelines

Returns a dictionary of all inspectable pipelines. The keys are stringified id of pipeline instances.

This includes user defined pipeline assigned to variables and anonymous pipelines with inspectable PCollections. If a user defined pipeline is not within the returned dict, it can be considered out of scope, and all resources and memory states related to it should be released.

list_inspectables(**kwargs)

get_val(identifier)

Retrieves the in memory object itself by identifier.

The retrieved object could be a pipeline or a pcollection. If the identifier is not recognized, return None. The identifier can refer to an anonymous pipeline and the object will still be retrieved.

get_pcoll_data(identifier, include_window_info=False)

Retrieves the json formatted PCollection data.

If no PCollection value can be retieved from the given identifier, an empty json string will be returned.

list_clusters(**kwargs)

get_cluster_master_url(identifier: str) → str

Returns the master_url corresponding to the obfuscated identifier.

apache_beam.runners.interactive.messaging.interactive_environment_inspector.inspect(ignore_synthetic=True)

Inspects current interactive environment to track metadata and values of pipelines and pcollections.

Each pipeline and pcollections tracked is given a unique identifier.

apache_beam.runners.interactive.messaging.interactive_environment_inspector.inspect_pipelines()

Inspects current interactive environment to track all pipelines assigned to variables. The keys are pipeline objects and values are pipeline names.

apache_beam.runners.interactive.messaging.interactive_environment_inspector.meta(name, val)

Generates meta data for the given name and value.

apache_beam.runners.interactive.messaging.interactive_environment_inspector.synthesize_pipeline_name(val)

Synthesizes a pipeline name for the given pipeline object.