apache_beam.transforms.stats module

This module has all statistic related transforms.

This ApproximateUnique class will be deprecated [1]. PLease look into using HLLCount in the zetasketch extension module [2].

[1] https://lists.apache.org/thread.html/501605df5027567099b81f18c080469661fb426 4a002615fa1510502%40%3Cdev.beam.apache.org%3E [2] https://beam.apache.org/releases/javadoc/2.16.0/org/apache/beam/sdk/extensio ns/zetasketch/HllCount.html

class apache_beam.transforms.stats.ApproximateQuantiles

Bases: object

PTransform for getting the idea of data distribution using approximate N-tile (e.g. quartiles, percentiles etc.) either globally or per-key.

Examples

in: list(range(101)), num_quantiles=5

out: [0, 25, 50, 75, 100]

in: [(i, 1 if i<10 else 1e-5) for i in range(101)], num_quantiles=5,

weighted=True

out: [0, 2, 5, 7, 100]

in: [list(range(10)), …, list(range(90, 101))], num_quantiles=5,

input_batched=True

out: [0, 25, 50, 75, 100]

in: [(list(range(10)), [1]*10), (list(range(10)), [0]*10), …,

(list(range(90, 101)), [0]*11)], num_quantiles=5, input_batched=True, weighted=True

out: [0, 2, 5, 7, 100]

class Globally(num_quantiles, key=None, reverse=False, weighted=False, input_batched=False)

Bases: PTransform

PTransform takes PCollection and returns a list whose single value is approximate N-tiles of the input collection globally.

Parameters:

  • num_quantiles – number of elements in the resulting quantiles values list.

  • key – (optional) Key is a mapping of elements to a comparable key, similar to the key argument of Python’s sorting methods.

  • reverse – (optional) whether to order things smallest to largest, rather than largest to smallest.

  • weighted – (optional) if set to True, the transform returns weighted quantiles. The input PCollection is then expected to contain tuples of input values with the corresponding weight.

  • input_batched – (optional) if set to True, the transform expects each element of input PCollection to be a batch, which is a list of elements for non-weighted case and a tuple of lists of elements and weights for weighted. Provides a way to accumulate multiple elements at a time more efficiently.

display_data()
expand(pcoll)
class PerKey(num_quantiles, key=None, reverse=False, weighted=False, input_batched=False)

Bases: PTransform

PTransform takes PCollection of KV and returns a list based on each key whose single value is list of approximate N-tiles of the input element of the key.

Parameters:

  • num_quantiles – number of elements in the resulting quantiles values list.

  • key – (optional) Key is a mapping of elements to a comparable key, similar to the key argument of Python’s sorting methods.

  • reverse – (optional) whether to order things smallest to largest, rather than largest to smallest.

  • weighted – (optional) if set to True, the transform returns weighted quantiles. The input PCollection is then expected to contain tuples of input values with the corresponding weight.

  • input_batched – (optional) if set to True, the transform expects each element of input PCollection to be a batch, which is a list of elements for non-weighted case and a tuple of lists of elements and weights for weighted. Provides a way to accumulate multiple elements at a time more efficiently.

display_data()
expand(pcoll)
class apache_beam.transforms.stats.ApproximateUnique

Bases: object

Hashes input elements and uses those to extrapolate the size of the entire set of hash values by assuming the rest of the hash values are as densely distributed as the sample space.

class Globally(size=None, error=None)

Bases: PTransform

Approximate.Globally approximate number of unique values

expand(pcoll)
class PerKey(size=None, error=None)

Bases: PTransform

Approximate.PerKey approximate number of unique values per key

expand(pcoll)
static parse_input_params(size=None, error=None)

Check if input params are valid and return sample size.

Parameters:

  • size – an int not smaller than 16, which we would use to estimate number of unique values.

  • error – max estimation error, which is a float between 0.01 and 0.50. If error is given, sample size will be calculated from error with _get_sample_size_from_est_error function.

Returns:

sample size

Raises:

ValueError: If both size and error are given, or neither is given, or values are out of range.