org.apache.beam.sdk.extensions.zetasketch

Class ApproximateCountDistinct

java.lang.Object

org.apache.beam.sdk.extensions.zetasketch.ApproximateCountDistinct

public class ApproximateCountDistinct
extends java.lang.Object

PTransforms for estimating the number of distinct elements in a PCollection, or the number of distinct values associated with each key in a PCollection of KVs.

We make use of the HllCount implementation for this transform. Please use HllCount directly if you need access to the sketches.

If the object is not one of Byte[] Integer Double String make use of ApproximateCountDistinct.Globally.via(org.apache.beam.sdk.transforms.ProcessFunction<T, java.lang.Long>) or ApproximateCountDistinct.PerKey.via(org.apache.beam.sdk.transforms.ProcessFunction<org.apache.beam.sdk.values.KV<K, V>, org.apache.beam.sdk.values.KV<K, java.lang.Long>>)

Examples

Example 1: Approximate Count of Ints PCollection<Integer> and specify precision

 p.apply("Int", Create.of(ints)).apply("IntHLL", ApproximateCountDistinct.globally()
   .withPercision(PRECISION));

 

Example 2: Approximate Count of Key Value PCollection<KV<Integer,Foo>>

 PCollection<KV<Integer, Long>> result =
   p.apply("Long", Create.of(longs)).apply("LongHLL", ApproximateCountDistinct.perKey());

 

Example 3: Approximate Count of Key Value PCollection<KV<Integer,Foo>>

 PCollection<KV<Integer, Foo>> approxResultInteger =
   p.apply("Int", Create.of(Foo))
     .apply("IntHLL", ApproximateCountDistinct.<Integer, KV<Integer, Integer>>perKey()
       .via(kv -> KV.of(kv.getKey(), (long) kv.getValue().hashCode())));
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ApproximateCountDistinct.Globally<T> PTransform for estimating the number of distinct elements in a PCollection.
static class	ApproximateCountDistinct.PerKey<K,V>

Constructor Summary

Constructors

Constructor and Description
ApproximateCountDistinct()

Method Summary

Modifier and Type	Method and Description
protected static <T> HllCount.Init.Builder<T>	builderForType(TypeDescriptor<T> input)
static <T> Combine.CombineFn<T,?,byte[]>	getUdaf(TypeDescriptor<T> input)
static <T> ApproximateCountDistinct.Globally<T>	globally()
static <K,V> ApproximateCountDistinct.PerKey<K,V>	perKey()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ApproximateCountDistinct

public ApproximateCountDistinct()

Method Detail

globally

public static <T> ApproximateCountDistinct.Globally<T> globally()

perKey

public static <K,V> ApproximateCountDistinct.PerKey<K,V> perKey()

getUdaf

public static <T> Combine.CombineFn<T,?,byte[]> getUdaf(TypeDescriptor<T> input)

builderForType

protected static <T> HllCount.Init.Builder<T> builderForType(TypeDescriptor<T> input)