Package org.apache.beam.sdk.transforms

Class Deduplicate

java.lang.Object
org.apache.beam.sdk.transforms.Deduplicate
public final class Deduplicate extends Object
A set of PTransforms which deduplicate input records over a time domain and threshold. Values in different windows will not be considered duplicates of each other. Deduplication is best effort.

Two values of type T are compared for equality not by regular Java Object.equals(java.lang.Object), but instead by first encoding each of the elements using the PCollection's Coder, and then comparing the encoded bytes. This admits efficient parallel evaluation.

These PTransforms are different then Distinct since Distinct guarantees uniqueness of values within a PCollection but may support a narrower set of windowing strategies or may delay when output is produced.

The durations specified may impose memory and/or storage requirements within a runner and care might need to be used to ensure that the deduplication time limit is long enough to remove duplicates but short enough to not cause performance problems within a runner. Each runner may provide an optimized implementation of their choice using the deduplication time domain and threshold specified.

Does not preserve any order the input PCollection might have had.

Example of use: 


 PCollection<String> words = ...;
 PCollection<String> deduplicatedWords =
     words.apply(Deduplicate.<String>values());