apache_beam.runners.dataflow.dataflow_metrics module

DataflowRunner implementation of MetricResults. It is in charge of responding to queries of current metrics by going to the dataflow service.

class apache_beam.runners.dataflow.dataflow_metrics.DataflowMetrics(dataflow_client=None, job_result=None, job_graph=None)

Bases: apache_beam.metrics.metric.MetricResults

Implementation of MetricResults class for the Dataflow runner.

Initialize the Dataflow metrics object.

Parameters:

• dataflow_client – apiclient.DataflowApplicationClient to interact with the dataflow service.
• job_result – DataflowPipelineResult with the state and id information of the job.
• job_graph – apiclient.Job instance to be able to translate between internal step names (e.g. “s2”), and user step names (e.g. “split”).

all_metrics(job_id=None)

Return all user and system metrics from the dataflow service.

query(filter=None)

apache_beam.runners.dataflow.dataflow_metrics.main(argv)

Print the metric results for a the dataflow –job_id and –project.

Instead of running an entire pipeline which takes several minutes, use this main method to display MetricResults for a specific –job_id and –project which takes only a few seconds.