Package org.apache.beam.sdk.metrics

Class Lineage

java.lang.Object

org.apache.beam.sdk.metrics.Lineage

public class Lineage
extends Object

Standard collection of metrics used to record source and sinks information for lineage tracking.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	Lineage.Type	Lineage metrics resource types.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	LINEAGE_NAMESPACE

Method Summary

Modifier and Type	Method	Description
void	add(Iterable<String> rollupSegments)	Adds the given fqn as lineage.
void	add(String system, @Nullable String subtype, Iterable<String> segments, @Nullable String lastSegmentSep)	Add a FQN (fully-qualified name) to Lineage.
void	add(String system, Iterable<String> segments)	Add a FQN (fully-qualified name) to Lineage.
void	add(String system, Iterable<String> segments, @Nullable String lastSegmentSep)	Add a FQN (fully-qualified name) to Lineage.
static Lineage	getSinks()	Lineage representing sinks.
static Lineage	getSources()	Lineage representing sources and optionally side inputs.
static Set<String>	query(MetricResults results, Lineage.Type type)	Query BoundedTrie metrics from MetricResults.
static Set<String>	query(MetricResults results, Lineage.Type type, String truncatedMarker)	Query BoundedTrie metrics from MetricResults.
static String	wrapSegment(String value)	Wrap segment to valid segment name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LINEAGE_NAMESPACE

public static final String LINEAGE_NAMESPACE

See Also:

• Constant Field Values

Method Details

getSources

public static Lineage getSources()

Lineage representing sources and optionally side inputs.

getSinks

public static Lineage getSinks()

Lineage representing sinks.

add

public void add(String system,
 @Nullable String subtype,
 Iterable<String> segments,
 @Nullable String lastSegmentSep)

Add a FQN (fully-qualified name) to Lineage. Segments will be processed via getFQNParts(java.lang.String, java.lang.String, java.lang.Iterable<java.lang.String>, java.lang.String).

add

public void add(String system,
 Iterable<String> segments,
 @Nullable String lastSegmentSep)

Add a FQN (fully-qualified name) to Lineage. Segments will be processed via getFQNParts(java.lang.String, java.lang.String, java.lang.Iterable<java.lang.String>, java.lang.String).

add

public void add(String system,
 Iterable<String> segments)

Add a FQN (fully-qualified name) to Lineage. Segments will be processed via getFQNParts(java.lang.String, java.lang.String, java.lang.Iterable<java.lang.String>, java.lang.String).

add

public void add(Iterable<String> rollupSegments)

Adds the given fqn as lineage.

Parameters:

rollupSegments - should be an iterable of strings whose concatenation is a valid Dataplex FQN which is already escaped.

In particular, this means they will often have trailing delimiters.

query

public static Set<String> query(MetricResults results,
 Lineage.Type type,
 String truncatedMarker)

Query BoundedTrie metrics from MetricResults.

Parameters:

results - FQNs from the result.

type - sources or sinks.

truncatedMarker - the marker to use to represent truncated FQNs.

Returns:

A flat representation of all FQNs. If the FQN was truncated then it has a trailing truncatedMarker.

query

public static Set<String> query(MetricResults results,
 Lineage.Type type)

Query BoundedTrie metrics from MetricResults.

Parameters:

results - FQNs from the result

type - sources or sinks

Returns:

A flat representation of all FQNs. If the FQN was truncated then it has a trailing '*'.

wrapSegment

@Internal
public static String wrapSegment(String value)

Wrap segment to valid segment name.

It escapes reserved characters

• Reserved characters are backtick, colon, whitespace (space, \t, \n) and dot.
• Only segments containing reserved characters must be escaped.
• Segments cannot be escaped partially (i.e. “bigquery:com`.`google.test”).
• Segments must be escaped using backticks (a.k.a. graves).
• Backticks must be escaped using backtick (i.e. bigquery:`test``test`) and the segment itself must be escaped as well.