apache_beam.transforms.combiners module

A library of basic combiner PTransform subclasses.

class apache_beam.transforms.combiners.Mean

Bases: object

Combiners for computing arithmetic means of elements.

class Globally(has_defaults=True)

Bases: apache_beam.transforms.combiners.CombinerWithoutDefaults

combiners.Mean.Globally computes the arithmetic mean of the elements.

expand(pcoll)

class PerKey(label=None)

Bases: apache_beam.transforms.ptransform.PTransform

combiners.Mean.PerKey finds the means of the values for each key.

expand(pcoll)

class apache_beam.transforms.combiners.MeanCombineFn(*unused_args, **unused_kwargs)

Bases: apache_beam.transforms.core.CombineFn

CombineFn for computing an arithmetic mean.

create_accumulator()

add_input(sum_count, element)

merge_accumulators(accumulators)

extract_output(sum_count)

for_input_type(input_type)

class apache_beam.transforms.combiners.Count

Bases: object

Combiners for counting elements.

class Globally(has_defaults=True)

Bases: apache_beam.transforms.combiners.CombinerWithoutDefaults

combiners.Count.Globally counts the total number of elements.

expand(pcoll)

class PerKey(label=None)

Bases: apache_beam.transforms.ptransform.PTransform

combiners.Count.PerKey counts how many elements each unique key has.

expand(pcoll)

class PerElement(label=None)

Bases: apache_beam.transforms.ptransform.PTransform

combiners.Count.PerElement counts how many times each element occurs.

expand(pcoll)

class apache_beam.transforms.combiners.CountCombineFn(*unused_args, **unused_kwargs)

Bases: apache_beam.transforms.core.CombineFn

CombineFn for computing PCollection size.

create_accumulator()

add_input(accumulator, element)

add_inputs(accumulator, elements)

merge_accumulators(accumulators)

extract_output(accumulator)

class apache_beam.transforms.combiners.Top

Bases: object

Combiners for obtaining extremal elements.

class Of(n, key=None, reverse=False)

Bases: apache_beam.transforms.combiners.CombinerWithoutDefaults

Returns the n greatest elements in the PCollection.

This transform will retrieve the n greatest elements in the PCollection to which it is applied, where “greatest” is determined by a function supplied as the key or reverse arguments.

Creates a global Top operation.

The arguments ‘key’ and ‘reverse’ may be passed as keyword arguments, and have the same meaning as for Python’s sort functions.

Parameters:

• n – number of elements to extract from pcoll.
• key – (optional) a mapping of elements to a comparable key, similar to the key argument of Python’s sorting methods.
• reverse – (optional) whether to order things smallest to largest, rather than largest to smallest

default_label()

expand(pcoll)

class PerKey(n, key=None, reverse=False)

Bases: apache_beam.transforms.ptransform.PTransform

Identifies the N greatest elements associated with each key.

This transform will produce a PCollection mapping unique keys in the input PCollection to the n greatest elements with which they are associated, where “greatest” is determined by a function supplied as the key or reverse arguments.

Creates a per-key Top operation.

The arguments ‘key’ and ‘reverse’ may be passed as keyword arguments, and have the same meaning as for Python’s sort functions.

Parameters:

• n – number of elements to extract from pcoll.
• key – (optional) a mapping of elements to a comparable key, similar to the key argument of Python’s sorting methods.
• reverse – (optional) whether to order things smallest to largest, rather than largest to smallest

default_label()

expand(pcoll)

Expands the transform.

Raises TypeCheckError: If the output type of the input PCollection is not compatible with Tuple[A, B].

Parameters:pcoll – PCollection to process Returns:the PCollection containing the result.

static Largest(n, has_defaults=True, key=None)

Obtain a list of the greatest N elements in a PCollection.

static Smallest(n, has_defaults=True, key=None)

Obtain a list of the least N elements in a PCollection.

static LargestPerKey(n, key=None)

Identifies the N greatest elements associated with each key.

static SmallestPerKey(n, *, key=None)

Identifies the N least elements associated with each key.

class apache_beam.transforms.combiners.TopCombineFn(n, key=None, reverse=False)

Bases: apache_beam.transforms.core.CombineFn

CombineFn doing the combining for all of the Top transforms.

This CombineFn uses a key or reverse operator to rank the elements.

Parameters:

• key – (optional) a mapping of elements to a comparable key, similar to the key argument of Python’s sorting methods.
• reverse – (optional) whether to order things smallest to largest, rather than largest to smallest

display_data()

create_accumulator(*args, **kwargs)

add_input(accumulator, element, *args, **kwargs)

merge_accumulators(accumulators, *args, **kwargs)

compact(accumulator, *args, **kwargs)

extract_output(accumulator, *args, **kwargs)

class apache_beam.transforms.combiners.Sample

Bases: object

Combiners for sampling n elements without replacement.

class FixedSizeGlobally(n)

Bases: apache_beam.transforms.combiners.CombinerWithoutDefaults

Sample n elements from the input PCollection without replacement.

expand(pcoll)

display_data()

default_label()

class FixedSizePerKey(n)

Bases: apache_beam.transforms.ptransform.PTransform

Sample n elements associated with each key without replacement.

expand(pcoll)

display_data()

default_label()

class apache_beam.transforms.combiners.SampleCombineFn(n)

Bases: apache_beam.transforms.core.CombineFn

CombineFn for all Sample transforms.

setup()

create_accumulator()

add_input(heap, element)

merge_accumulators(heaps)

compact(heap)

extract_output(heap)

teardown()

class apache_beam.transforms.combiners.ToList(has_defaults=True)

Bases: apache_beam.transforms.combiners.CombinerWithoutDefaults

A global CombineFn that condenses a PCollection into a single list.

expand(pcoll)

class apache_beam.transforms.combiners.ToListCombineFn(*unused_args, **unused_kwargs)

Bases: apache_beam.transforms.core.CombineFn

CombineFn for to_list.

create_accumulator()

add_input(accumulator, element)

merge_accumulators(accumulators)

extract_output(accumulator)

class apache_beam.transforms.combiners.ToDict(has_defaults=True)

Bases: apache_beam.transforms.combiners.CombinerWithoutDefaults

A global CombineFn that condenses a PCollection into a single dict.

PCollections should consist of 2-tuples, notionally (key, value) pairs. If multiple values are associated with the same key, only one of the values will be present in the resulting dict.

expand(pcoll)

class apache_beam.transforms.combiners.ToDictCombineFn(*unused_args, **unused_kwargs)

Bases: apache_beam.transforms.core.CombineFn

CombineFn for to_dict.

create_accumulator()

add_input(accumulator, element)

merge_accumulators(accumulators)

extract_output(accumulator)

class apache_beam.transforms.combiners.ToSet(has_defaults=True)

Bases: apache_beam.transforms.combiners.CombinerWithoutDefaults

A global CombineFn that condenses a PCollection into a set.

expand(pcoll)

class apache_beam.transforms.combiners.ToSetCombineFn(*unused_args, **unused_kwargs)

Bases: apache_beam.transforms.core.CombineFn

CombineFn for ToSet.

create_accumulator()

add_input(accumulator, element)

merge_accumulators(accumulators)

extract_output(accumulator)

class apache_beam.transforms.combiners.Latest

Bases: object

Combiners for computing the latest element

class Globally(has_defaults=True)

Bases: apache_beam.transforms.combiners.CombinerWithoutDefaults

Compute the element with the latest timestamp from a PCollection.

static add_timestamp(element, timestamp=TimestampParam)

expand(pcoll)

class PerKey(label=None)

Bases: apache_beam.transforms.ptransform.PTransform

Compute elements with the latest timestamp for each key from a keyed PCollection

static add_timestamp(element, timestamp=TimestampParam)

expand(pcoll)

class apache_beam.transforms.combiners.LatestCombineFn(*unused_args, **unused_kwargs)

Bases: apache_beam.transforms.core.CombineFn

CombineFn to get the element with the latest timestamp from a PCollection.

create_accumulator()

add_input(accumulator, element)

merge_accumulators(accumulators)

extract_output(accumulator)