apache_beam.metrics package

Submodules

apache_beam.metrics.cells module

This file contains metric cell classes. A metric cell is used to accumulate in-memory changes to a metric. It represents a specific metric in a single context.

Cells depend on a ‘dirty-bit’ in the CellCommitState class that tracks whether a cell’s updates have been committed.

class apache_beam.metrics.cells.DistributionResult(data)

Bases: object

The result of a Distribution metric.

count

max

mean

Returns the float mean of the distribution.

If the distribution contains no elements, it returns None.

min

sum

apache_beam.metrics.execution module

This module is for internal use only; no backwards-compatibility guarantees.

The classes in this file keep shared state, and organize metrics information.

Available classes:

• MetricKey - Internal key for a metric.
• MetricResult - Current status of a metric’s updates/commits.

• _MetricsEnvironment - Keeps track of MetricsContainer and other metrics

  information for every single execution working thread.

• MetricsContainer - Holds the metrics of a single step and a single

  unit-of-commit (bundle).

class apache_beam.metrics.execution.MetricKey(step, metric)

Bases: object

Key used to identify instance of metric cell.

Metrics are internally keyed by the step name they associated with and the name of the metric.

class apache_beam.metrics.execution.MetricResult(key, committed, attempted)

Bases: object

Keeps track of the status of a metric within a single bundle.

It contains the physical and logical updates to the metric. Physical updates are updates that have not necessarily been committed, but that have been made during pipeline execution. Logical updates are updates that have been committed.

key

A MetricKey that identifies the metric and bundle of this result.

committed

The committed updates of the metric. This attribute’s type is that of the underlying cell data (e.g. int, DistributionData).

attempted

The logical updates of the metric. This attribute’s type is that of the underlying cell data (e.g. int, DistributionData).

class apache_beam.metrics.execution.MetricUpdates(counters=None, distributions=None)

Bases: object

Contains updates for several metrics.

A metric update is an object containing information to update a metric. For Distribution metrics, it is DistributionData, and for Counter metrics, it’s an int.

class apache_beam.metrics.execution.MetricsContainer(step_name)

Bases: object

Holds the metrics of a single step and a single bundle.

get_counter(metric_name)

get_cumulative()

Return MetricUpdates with cumulative values of all metrics in container.

This returns all the cumulative values for all metrics regardless of whether they have been committed or not.

get_distribution(metric_name)

get_updates()

Return cumulative values of metrics that changed since the last commit.

This returns all the cumulative values for all metrics only if their state prior to the function call was COMMITTING or DIRTY.

class apache_beam.metrics.execution.ScopedMetricsContainer(container=None)

Bases: object

enter()

exit()

apache_beam.metrics.metric module

User-facing classes for Metrics API.

The classes in this file allow users to define and use metrics to be collected and displayed as part of their pipeline execution.

• Metrics - This class lets pipeline and transform writers create and access

  metric objects such as counters, distributions, etc.

class apache_beam.metrics.metric.Metrics

Bases: object

Lets users create/access metric objects during pipeline execution.

class DelegatingCounter(metric_name)

Bases: apache_beam.metrics.metricbase.Counter

inc(n=1)

class DelegatingDistribution(metric_name)

Bases: apache_beam.metrics.metricbase.Distribution

update(value)

static counter(namespace, name)

Obtains or creates a Counter metric.

Parameters:

• namespace – A class or string that gives the namespace to a metric
• name – A string that gives a unique name to a metric

Returns:

A Counter object.

static distribution(namespace, name)

Obtains or creates a Distribution metric.

Distribution metrics are restricted to integer-only distributions.

Parameters:

• namespace – A class or string that gives the namespace to a metric
• name – A string that gives a unique name to a metric

Returns:

A Distribution object.

static get_namespace(namespace)

class apache_beam.metrics.metric.MetricsFilter

Bases: object

Simple object to filter metrics results.

This class is experimental. No backwards-compatibility guarantees.

If filters by matching a result’s step-namespace-name with three internal sets. No execution/matching logic is added to this object, so that it may be used to construct arguments as an RPC request. It is left for runners to implement matching logic by themselves.

names

namespaces

steps

with_name(name)

with_names(names)

with_namespace(namespace)

with_namespaces(namespaces)

with_step(step)

with_steps(steps)

apache_beam.metrics.metricbase module

The classes in this file are interfaces for metrics. They are not intended to be subclassed or created directly by users. To work with and access metrics, users should use the classes and methods exposed in metric.py.

Available classes:

• Metric - Base interface of a metrics object.

• Counter - Counter metric interface. Allows a count to be incremented or

  decremented during pipeline execution.

• Distribution - Distribution Metric interface. Allows statistics about the

  distribution of a variable to be collected during pipeline execution.

• MetricName - Namespace and name used to refer to a Metric.

class apache_beam.metrics.metricbase.Metric

Bases: object

Base interface of a metric object.

class apache_beam.metrics.metricbase.Counter

Bases: apache_beam.metrics.metricbase.Metric

Counter metric interface. Allows a count to be incremented/decremented during pipeline execution.

dec(n=1)

inc(n=1)

class apache_beam.metrics.metricbase.Distribution

Bases: apache_beam.metrics.metricbase.Metric

Distribution Metric interface. Allows statistics about the distribution of a variable to be collected during pipeline execution.

update(value)

class apache_beam.metrics.metricbase.MetricName(namespace, name)

Bases: object

The name of a metric.

The name of a metric consists of a namespace and a name. The namespace allows grouping related metrics together and also prevents collisions between multiple metrics of the same name.

Module contents