apache_beam.metrics.metric module

User-facing classes for Metrics API.

The classes in this file allow users to define and use metrics to be collected and displayed as part of their pipeline execution.

• Metrics - This class lets pipeline and transform writers create and access

  metric objects such as counters, distributions, etc.

class apache_beam.metrics.metric.Metrics

Bases: object

Lets users create/access metric objects during pipeline execution.

static get_namespace(namespace: Type | str) → str

static counter(namespace: Type | str, name: str) → DelegatingCounter

Obtains or creates a Counter metric.

Parameters:

• namespace – A class or string that gives the namespace to a metric

• name – A string that gives a unique name to a metric

Returns:

A Counter object.

static distribution(namespace: Type | str, name: str) → DelegatingDistribution

Obtains or creates a Distribution metric.

Distribution metrics are restricted to integer-only distributions.

Parameters:

• namespace – A class or string that gives the namespace to a metric

• name – A string that gives a unique name to a metric

Returns:

A Distribution object.

static gauge(namespace: Type | str, name: str) → DelegatingGauge

Obtains or creates a Gauge metric.

Gauge metrics are restricted to integer-only values.

Parameters:

• namespace – A class or string that gives the namespace to a metric

• name – A string that gives a unique name to a metric

Returns:

A Distribution object.

static string_set(namespace: Type | str, name: str) → DelegatingStringSet

Obtains or creates a String set metric.

String set metrics are restricted to string values.

Parameters:

• namespace – A class or string that gives the namespace to a metric

• name – A string that gives a unique name to a metric

Returns:

A StringSet object.

static bounded_trie(namespace: Type | str, name: str) → DelegatingBoundedTrie

Obtains or creates a Bounded Trie metric.

Parameters:

• namespace – A class or string that gives the namespace to a metric

• name – A string that gives a unique name to a metric

Returns:

A BoundedTrie object.

class DelegatingCounter(metric_name: MetricName, process_wide: bool = False)

Bases: Counter

Metrics Counter that Delegates functionality to MetricsEnvironment.

class DelegatingDistribution(metric_name: MetricName)

Bases: Distribution

Metrics Distribution Delegates functionality to MetricsEnvironment.

class DelegatingGauge(metric_name: MetricName)

Bases: Gauge

Metrics Gauge that Delegates functionality to MetricsEnvironment.

class DelegatingStringSet(metric_name: MetricName)

Bases: StringSet

Metrics StringSet that Delegates functionality to MetricsEnvironment.

class DelegatingBoundedTrie(metric_name: MetricName)

Bases: BoundedTrie

Metrics StringSet that Delegates functionality to MetricsEnvironment.

class apache_beam.metrics.metric.MetricsFilter

Bases: object

Simple object to filter metrics results.

If filters by matching a result’s step-namespace-name with three internal sets. No execution/matching logic is added to this object, so that it may be used to construct arguments as an RPC request. It is left for runners to implement matching logic by themselves.

Note: This class only supports user defined metrics.

property steps: FrozenSet[str]

property names: FrozenSet[str]

property namespaces: FrozenSet[str]

with_metric(metric: Metric) → MetricsFilter

with_name(name: str) → MetricsFilter

with_names(names: Iterable[str]) → MetricsFilter

with_namespace(namespace: Type | str) → MetricsFilter

with_namespaces(namespaces: Iterable[Type | str]) → MetricsFilter

with_step(step: str) → MetricsFilter

with_steps(steps: Iterable[str]) → MetricsFilter

class apache_beam.metrics.metric.Lineage(label: str)

Bases: object

Standard collection of metrics used to record source and sinks information for lineage tracking.

Create a Lineage with valid label (SOURCE or SINK)

LINEAGE_NAMESPACE = 'lineage'

SOURCE = 'sources_v2'

SINK = 'sinks_v2'

classmethod sources() → Lineage

classmethod sinks() → Lineage

static wrap_segment(segment: str) → str

Wrap segment to valid segment name.

Specifically, If there are reserved chars (colon, whitespace, dot), escape with backtick. If the segment is already wrapped, return the original.

static get_fq_name(system: str, *segments: str, subtype: str | None = None) → str

Assemble fully qualified name (FQN). Format:

• system:segment1.segment2

• system:subtype:segment1.segment2

• system:`segment1.with.dots:colons.segment2`

This helper method is for internal and testing usage only.

add(system: str, *segments: str, subtype: str | None = None, last_segment_sep: str | None = None) → None

Adds the given details as Lineage.

For asset level lineage the resource location should be specified as Dataplex FQN, see https://cloud.google.com/data-catalog/docs/fully-qualified-names

Example of adding FQN components:

• add(“system”, “segment1”, “segment2”)

• add(“system”, “segment1”, “segment2”, subtype=”subtype”)

Example of adding a FQN:

• add(“system:segment1.segment2”)

• add(“system:subtype:segment1.segment2”)

The first positional argument serves as system, if full segments are provided, or the full FQN if it is provided as a single argument.

add_raw(*rollup_segments: str) → None

Adds the given fqn as lineage.

rollup_segments should be an iterable of strings whose concatenation is a valid Dataplex FQN. In particular, this means they will often have trailing delimiters.

static query(results: MetricResults, label: str, truncated_marker: str = '*') → Set[str]