org.apache.beam.sdk.transforms

Class Deduplicate

java.lang.Object

org.apache.beam.sdk.transforms.Deduplicate

public final class Deduplicate
extends java.lang.Object

A set of PTransforms which deduplicate input records over a time domain and threshold. Values in different windows will not be considered duplicates of each other. Deduplication is best effort.

Two values of type T are compared for equality not by regular Java Object.equals(java.lang.Object), but instead by first encoding each of the elements using the PCollection's Coder, and then comparing the encoded bytes. This admits efficient parallel evaluation.

These PTransforms are different then Distinct since Distinct guarantees uniqueness of values within a PCollection but may support a narrower set of windowing strategies or may delay when output is produced.

The durations specified may impose memory and/or storage requirements within a runner and care might need to be used to ensure that the deduplication time limit is long enough to remove duplicates but short enough to not cause performance problems within a runner. Each runner may provide an optimized implementation of their choice using the deduplication time domain and threshold specified.

Does not preserve any order the input PCollection might have had.

Example of use:

 PCollection<String> words = ...;
 PCollection<String> deduplicatedWords =
     words.apply(Deduplicate.<String>values());
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Deduplicate.KeyedValues<K,V> Deduplicates keyed values using the key over a specified time domain and threshold.
static class	Deduplicate.Values<T> Deduplicates values over a specified time domain and threshold.
static class	Deduplicate.WithRepresentativeValues<T,IdT> A PTransform that uses a SerializableFunction to obtain a representative value for each input element used for deduplication.

Field Summary

Fields

Modifier and Type	Field and Description
static Duration	DEFAULT_DURATION The default duration is 10 mins.
static TimeDomain	DEFAULT_TIME_DOMAIN The default is the processing time domain.

Method Summary

Modifier and Type	Method and Description
static <K,V> Deduplicate.KeyedValues<K,V>	keyedValues() Returns a deduplication transform that deduplicates keyed values using the key for up to 10 mins within the processing time domain.
static <T> Deduplicate.Values<T>	values() Returns a deduplication transform that deduplicates values for up to 10 mins within the processing time domain.
static <T,IdT> Deduplicate.WithRepresentativeValues<T,IdT>	withRepresentativeValueFn(SerializableFunction<T,IdT> representativeValueFn) Returns a deduplication transform that deduplicates values using the supplied representative value for up to 10 mins within the processing time domain.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_TIME_DOMAIN

public static final TimeDomain DEFAULT_TIME_DOMAIN

The default is the processing time domain.

DEFAULT_DURATION

public static final Duration DEFAULT_DURATION

The default duration is 10 mins.

Method Detail

values

public static <T> Deduplicate.Values<T> values()

Returns a deduplication transform that deduplicates values for up to 10 mins within the processing time domain.

keyedValues

public static <K,V> Deduplicate.KeyedValues<K,V> keyedValues()

Returns a deduplication transform that deduplicates keyed values using the key for up to 10 mins within the processing time domain.

withRepresentativeValueFn

public static <T,IdT> Deduplicate.WithRepresentativeValues<T,IdT> withRepresentativeValueFn(SerializableFunction<T,IdT> representativeValueFn)

Returns a deduplication transform that deduplicates values using the supplied representative value for up to 10 mins within the processing time domain.