apache_beam.runners.dataflow.dataflow_metrics module

DataflowRunner implementation of MetricResults. It is in charge of responding to queries of current metrics by going to the dataflow service.

class apache_beam.runners.dataflow.dataflow_metrics.DataflowMetrics(dataflow_client=None, job_result=None, job_graph=None)[source]

Bases: apache_beam.metrics.metric.MetricResults

Implementation of MetricResults class for the Dataflow runner.

Initialize the Dataflow metrics object.

Parameters:
  • dataflow_client – apiclient.DataflowApplicationClient to interact with the dataflow service.
  • job_result – DataflowPipelineResult with the state and id information of the job.
  • job_graph – apiclient.Job instance to be able to translate between internal step names (e.g. “s2”), and user step names (e.g. “split”).
all_metrics(job_id=None)[source]

Return all user and system metrics from the dataflow service.

query(filter=None)[source]
apache_beam.runners.dataflow.dataflow_metrics.main(argv)[source]

Print the metric results for the dataflow –job_id and –project.

Instead of running an entire pipeline which takes several minutes, use this main method to display MetricResults for a specific –job_id and –project which takes only a few seconds.