apache_beam.runners.interactive.dataproc.types module

class apache_beam.runners.interactive.dataproc.types.ClusterMetadata(project_id: Optional[str] = None, region: Optional[str] = 'us-central1', cluster_name: Optional[str] = <factory>, subnetwork: Optional[str] = None, num_workers: Optional[int] = None, machine_type: Optional[int] = None, master_url: Optional[str] = None, dashboard: Optional[str] = None)

Bases: object

Metadata of a provisioned worker cluster that executes Beam pipelines.

Apache Beam supports running Beam pipelines on different runners provisioned in different setups based on the runner and pipeline options associated with each pipeline. To provide similar portability features, Interactive Beam automatically extracts such ClusterMetadata information from pipeline options of a pipeline in the REPL context and provision suitable clusters to execute the pipeline. The lifecyle of the clusters is managed by Interactive Beam and the user doesn not need to interact with it.

It’s not recommended to build this ClusterMetadata from raw values nor use it to interact with the cluster management logic directly.

Interactive Beam now supports:

1. Runner: FlinkRunner; Setup: on Google Cloud with Flink on Dataproc.

project_id = None

region = 'us-central1'

subnetwork = None

num_workers = None

machine_type = None

master_url = None

dashboard = None

reset_name()