apache_beam.metrics.metric module

User-facing classes for Metrics API.

The classes in this file allow users to define and use metrics to be collected and displayed as part of their pipeline execution.

  • Metrics - This class lets pipeline and transform writers create and access
    metric objects such as counters, distributions, etc.
class apache_beam.metrics.metric.Metrics[source]

Bases: object

Lets users create/access metric objects during pipeline execution.

static get_namespace(namespace: Union[Type[CT_co], str]) → str[source]
static counter(namespace: Union[Type[CT_co], str], name: str) → apache_beam.metrics.metric.Metrics.DelegatingCounter[source]

Obtains or creates a Counter metric.

Parameters:
  • namespace – A class or string that gives the namespace to a metric
  • name – A string that gives a unique name to a metric
Returns:

A Counter object.

static distribution(namespace: Union[Type[CT_co], str], name: str) → apache_beam.metrics.metric.Metrics.DelegatingDistribution[source]

Obtains or creates a Distribution metric.

Distribution metrics are restricted to integer-only distributions.

Parameters:
  • namespace – A class or string that gives the namespace to a metric
  • name – A string that gives a unique name to a metric
Returns:

A Distribution object.

static gauge(namespace: Union[Type[CT_co], str], name: str) → apache_beam.metrics.metric.Metrics.DelegatingGauge[source]

Obtains or creates a Gauge metric.

Gauge metrics are restricted to integer-only values.

Parameters:
  • namespace – A class or string that gives the namespace to a metric
  • name – A string that gives a unique name to a metric
Returns:

A Distribution object.

static string_set(namespace: Union[Type[CT_co], str], name: str) → apache_beam.metrics.metric.Metrics.DelegatingStringSet[source]

Obtains or creates a String set metric.

String set metrics are restricted to string values.

Parameters:
  • namespace – A class or string that gives the namespace to a metric
  • name – A string that gives a unique name to a metric
Returns:

A StringSet object.

class DelegatingCounter(metric_name: apache_beam.metrics.metricbase.MetricName, process_wide: bool = False)[source]

Bases: apache_beam.metrics.metricbase.Counter

Metrics Counter that Delegates functionality to MetricsEnvironment.

class DelegatingDistribution(metric_name: apache_beam.metrics.metricbase.MetricName)[source]

Bases: apache_beam.metrics.metricbase.Distribution

Metrics Distribution Delegates functionality to MetricsEnvironment.

class DelegatingGauge(metric_name: apache_beam.metrics.metricbase.MetricName)[source]

Bases: apache_beam.metrics.metricbase.Gauge

Metrics Gauge that Delegates functionality to MetricsEnvironment.

class DelegatingStringSet(metric_name: apache_beam.metrics.metricbase.MetricName)[source]

Bases: apache_beam.metrics.metricbase.StringSet

Metrics StringSet that Delegates functionality to MetricsEnvironment.

class apache_beam.metrics.metric.MetricsFilter[source]

Bases: object

Simple object to filter metrics results.

If filters by matching a result’s step-namespace-name with three internal sets. No execution/matching logic is added to this object, so that it may be used to construct arguments as an RPC request. It is left for runners to implement matching logic by themselves.

Note: This class only supports user defined metrics.

steps
names
namespaces
with_metric(metric: Metric) → MetricsFilter[source]
with_name(name: str) → apache_beam.metrics.metric.MetricsFilter[source]
with_names(names: Iterable[str]) → apache_beam.metrics.metric.MetricsFilter[source]
with_namespace(namespace: Union[Type[CT_co], str]) → apache_beam.metrics.metric.MetricsFilter[source]
with_namespaces(namespaces: Iterable[Union[Type[CT_co], str]]) → apache_beam.metrics.metric.MetricsFilter[source]
with_step(step: str) → apache_beam.metrics.metric.MetricsFilter[source]
with_steps(steps: Iterable[str]) → apache_beam.metrics.metric.MetricsFilter[source]
class apache_beam.metrics.metric.Lineage(label: str)[source]

Bases: object

Standard collection of metrics used to record source and sinks information for lineage tracking.

Create a Lineage with valid label (SOURCE or SINK)

LINEAGE_NAMESPACE = 'lineage'
SOURCE = 'sources'
SINK = 'sinks'
classmethod sources() → apache_beam.metrics.metric.Lineage[source]
classmethod sinks() → apache_beam.metrics.metric.Lineage[source]
static wrap_segment(segment: str) → str[source]

Wrap segment to valid segment name.

Specifically, If there are reserved chars (colon, whitespace, dot), escape with backtick. If the segment is already wrapped, return the original.

static get_fq_name(system: str, *segments, route: Optional[str] = None) → str[source]

Assemble fully qualified name (FQN). Format:

  • system:segment1.segment2
  • system:routine:segment1.segment2
  • system:`segment1.with.dots:clons.segment2`

This helper method is for internal and testing usage only.

add(system: str, *segments, route: Optional[str] = None) → None[source]
static query(results: apache_beam.metrics.metric.MetricResults, label: str) → Set[str][source]