org.apache.beam.sdk.metrics

Class Lineage

java.lang.Object

org.apache.beam.sdk.metrics.Lineage

public class Lineage
extends java.lang.Object

Standard collection of metrics used to record source and sinks information for lineage tracking.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Lineage.Type Lineage metrics resource types.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	LINEAGE_NAMESPACE

Method Summary

Modifier and Type	Method and Description
void	add(java.lang.Iterable<java.lang.String> rollupSegments) Adds the given fqn as lineage.
void	add(java.lang.String system, java.lang.Iterable<java.lang.String> segments) Add a FQN (fully-qualified name) to Lineage.
void	add(java.lang.String system, java.lang.Iterable<java.lang.String> segments, @Nullable java.lang.String lastSegmentSep) Add a FQN (fully-qualified name) to Lineage.
void	add(java.lang.String system, @Nullable java.lang.String subtype, java.lang.Iterable<java.lang.String> segments, @Nullable java.lang.String lastSegmentSep) Add a FQN (fully-qualified name) to Lineage.
static Lineage	getSinks() Lineage representing sinks.
static Lineage	getSources() Lineage representing sources and optionally side inputs.
static java.util.Set<java.lang.String>	query(MetricResults results, Lineage.Type type) Query BoundedTrie metrics from MetricResults.
static java.util.Set<java.lang.String>	query(MetricResults results, Lineage.Type type, java.lang.String truncatedMarker) Query BoundedTrie metrics from MetricResults.
static java.lang.String	wrapSegment(java.lang.String value) Wrap segment to valid segment name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LINEAGE_NAMESPACE

public static final java.lang.String LINEAGE_NAMESPACE

See Also:

Constant Field Values

Method Detail

getSources

public static Lineage getSources()

Lineage representing sources and optionally side inputs.

getSinks

public static Lineage getSinks()

Lineage representing sinks.

add

public void add(java.lang.String system,
                @Nullable java.lang.String subtype,
                java.lang.Iterable<java.lang.String> segments,
                @Nullable java.lang.String lastSegmentSep)

Add a FQN (fully-qualified name) to Lineage. Segments will be processed via getFQNParts(java.lang.String, java.lang.String, java.lang.Iterable<java.lang.String>, java.lang.String).

add

public void add(java.lang.String system,
                java.lang.Iterable<java.lang.String> segments,
                @Nullable java.lang.String lastSegmentSep)

Add a FQN (fully-qualified name) to Lineage. Segments will be processed via getFQNParts(java.lang.String, java.lang.String, java.lang.Iterable<java.lang.String>, java.lang.String).

add

public void add(java.lang.String system,
                java.lang.Iterable<java.lang.String> segments)

Add a FQN (fully-qualified name) to Lineage. Segments will be processed via getFQNParts(java.lang.String, java.lang.String, java.lang.Iterable<java.lang.String>, java.lang.String).

add

public void add(java.lang.Iterable<java.lang.String> rollupSegments)

Adds the given fqn as lineage.

Parameters:

rollupSegments - should be an iterable of strings whose concatenation is a valid Dataplex FQN which is already escaped.

In particular, this means they will often have trailing delimiters.

query

public static java.util.Set<java.lang.String> query(MetricResults results,
                                                    Lineage.Type type,
                                                    java.lang.String truncatedMarker)

Query BoundedTrie metrics from MetricResults.

Parameters:

results - FQNs from the result.

type - sources or sinks.

truncatedMarker - the marker to use to represent truncated FQNs.

Returns:

A flat representation of all FQNs. If the FQN was truncated then it has a trailing truncatedMarker.

query

public static java.util.Set<java.lang.String> query(MetricResults results,
                                                    Lineage.Type type)

Query BoundedTrie metrics from MetricResults.

Parameters:

results - FQNs from the result

type - sources or sinks

Returns:

A flat representation of all FQNs. If the FQN was truncated then it has a trailing '*'.

wrapSegment

@Internal
public static java.lang.String wrapSegment(java.lang.String value)

Wrap segment to valid segment name.

It escapes reserved characters

• Reserved characters are backtick, colon, whitespace (space, \t, \n) and dot.
• Only segments containing reserved characters must be escaped.
• Segments cannot be escaped partially (i.e. “bigquery:com`.`google.test”).
• Segments must be escaped using backticks (a.k.a. graves).
• Backticks must be escaped using backtick (i.e. bigquery:`test``test`) and the segment itself must be escaped as well.