Package org.apache.beam.sdk.transforms

Class Distinct<T>

java.lang.Object
org.apache.beam.sdk.transforms.PTransform<PCollection<T>,PCollection<T>>
org.apache.beam.sdk.transforms.Distinct<T>
Type Parameters:
T - the type of the elements of the input and output PCollections
All Implemented Interfaces:
Serializable, HasDisplayData
public class Distinct<T> extends PTransform<PCollection<T>,PCollection<T>>
Distinct<T> takes a PCollection<T> and returns a PCollection<T> that has all distinct elements of the input. Thus, each element is unique within each window.

Two values of type T are compared for equality not by regular Java Object.equals(java.lang.Object), but instead by first encoding each of the elements using the PCollection's Coder, and then comparing the encoded bytes. This admits efficient parallel evaluation.

Optionally, a function may be provided that maps each element to a representative value. In this case, two elements will be considered duplicates if they have equal representative values, with equality being determined as above.

By default, the Coder of the output PCollection is the same as the Coder of the input PCollection.

Each output element is in the same window as its corresponding input element, and has the timestamp of the end of that window. The output PCollection has the same WindowFn as the input.

Does not preserve any order the input PCollection might have had.

Example of use: 


 PCollection<String> words = ...;
 PCollection<String> uniqueWords =
     words.apply(Distinct.<String>create());
See Also:

  • Constructor Details

    • Distinct

      public Distinct()

  • Method Details

    • create

      public static <T> Distinct<T> create()
      Returns a Distinct<T> PTransform.
      Type Parameters:
      T - the type of the elements of the input and output PCollections

    • withRepresentativeValueFn

      public static <T, IdT> Distinct.WithRepresentativeValues<T,IdT> withRepresentativeValueFn(SerializableFunction<T,IdT> fn)
      Returns a Distinct<T, IdT> PTransform.
      Type Parameters:
      T - the type of the elements of the input and output PCollections
      IdT - the type of the representative value used to dedup

    • expand

      public PCollection<T> expand(PCollection<T> in)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PCollection<T>,PCollection<T>>