apache_beam.runners.dataflow.dataflow_metrics module

DataflowRunner implementation of MetricResults. It is in charge of responding to queries of current metrics by going to the dataflow service.

class apache_beam.runners.dataflow.dataflow_metrics.DataflowMetrics(dataflow_client=None, job_result=None, job_graph=None)[source]

Bases: apache_beam.metrics.metric.MetricResults

Implementation of MetricResults class for the Dataflow runner.

Initialize the Dataflow metrics object.

  • dataflow_client – apiclient.DataflowApplicationClient to interact with the dataflow service.
  • job_result – DataflowPipelineResult with the state and id information of the job.
  • job_graph – apiclient.Job instance to be able to translate between internal step names (e.g. “s2”), and user step names (e.g. “split”).

Return all user and system metrics from the dataflow service.


Print the metric results for the dataflow –job_id and –project.

Instead of running an entire pipeline which takes several minutes, use this main method to display MetricResults for a specific –job_id and –project which takes only a few seconds.