apache_beam.transforms.combiners module

A library of basic combiner PTransform subclasses.

class apache_beam.transforms.combiners.Count

Bases: object

Combiners for counting elements.

class Globally(has_defaults=True)

Bases: CombinerWithoutDefaults

combiners.Count.Globally counts the total number of elements.

expand(pcoll)

class PerKey(label: str | None = None)

Bases: PTransform

combiners.Count.PerKey counts how many elements each unique key has.

expand(pcoll)

class PerElement(label: str | None = None)

Bases: PTransform

combiners.Count.PerElement counts how many times each element occurs.

expand(pcoll)

class apache_beam.transforms.combiners.Mean

Bases: object

Combiners for computing arithmetic means of elements.

class Globally(has_defaults=True)

Bases: CombinerWithoutDefaults

combiners.Mean.Globally computes the arithmetic mean of the elements.

expand(pcoll)

class PerKey(label: str | None = None)

Bases: PTransform

combiners.Mean.PerKey finds the means of the values for each key.

expand(pcoll)

class apache_beam.transforms.combiners.Sample

Bases: object

Combiners for sampling n elements without replacement.

class FixedSizeGlobally(n)

Bases: CombinerWithoutDefaults

Sample n elements from the input PCollection without replacement.

expand(pcoll)

display_data()

default_label()

class FixedSizePerKey(n)

Bases: PTransform

Sample n elements associated with each key without replacement.

expand(pcoll)

display_data()

default_label()

class Any(n)

Bases: PTransform

Returns up to n arbitrary elements from the input PCollection.

This is the Python equivalent of Java’s Sample.any. Unlike FixedSizeGlobally it does not sample uniformly at random, and it returns the selected elements rather than a single list. If the input has fewer than n elements, all of them are returned.

expand(pcoll)

display_data()

default_label()

class apache_beam.transforms.combiners.Top

Bases: object

Combiners for obtaining extremal elements.

class Of(n, key=None, reverse=False)

Bases: CombinerWithoutDefaults

Returns the n greatest elements in the PCollection.

This transform will retrieve the n greatest elements in the PCollection to which it is applied, where “greatest” is determined by a function supplied as the key or reverse arguments.

Creates a global Top operation.

The arguments ‘key’ and ‘reverse’ may be passed as keyword arguments, and have the same meaning as for Python’s sort functions.

Parameters:

• n – number of elements to extract from pcoll.

• key – (optional) a mapping of elements to a comparable key, similar to the key argument of Python’s sorting methods.

• reverse – (optional) whether to order things smallest to largest, rather than largest to smallest

default_label()

expand(pcoll)

class PerKey(n, key=None, reverse=False)

Bases: PTransform

Identifies the N greatest elements associated with each key.

This transform will produce a PCollection mapping unique keys in the input PCollection to the n greatest elements with which they are associated, where “greatest” is determined by a function supplied as the key or reverse arguments.

Creates a per-key Top operation.

The arguments ‘key’ and ‘reverse’ may be passed as keyword arguments, and have the same meaning as for Python’s sort functions.

Parameters:

• n – number of elements to extract from pcoll.

• key – (optional) a mapping of elements to a comparable key, similar to the key argument of Python’s sorting methods.

• reverse – (optional) whether to order things smallest to largest, rather than largest to smallest

default_label()

expand(pcoll)

Expands the transform.

Raises TypeCheckError: If the output type of the input PCollection is not compatible with tuple[A, B].

Parameters:

pcoll – PCollection to process

Returns:

the PCollection containing the result.

static Largest(n, has_defaults=True, key=None)

Obtain a list of the greatest N elements in a PCollection.

static Smallest(n, has_defaults=True, key=None)

Obtain a list of the least N elements in a PCollection.

static LargestPerKey(n, key=None)

Identifies the N greatest elements associated with each key.

static SmallestPerKey(n, *, key=None)

Identifies the N least elements associated with each key.

class apache_beam.transforms.combiners.ToDict(has_defaults=True)

Bases: CombinerWithoutDefaults

A global CombineFn that condenses a PCollection into a single dict.

PCollections should consist of 2-tuples, notionally (key, value) pairs. If multiple values are associated with the same key, only one of the values will be present in the resulting dict.

expand(pcoll)

class apache_beam.transforms.combiners.ToList(has_defaults=True)

Bases: CombinerWithoutDefaults

A global CombineFn that condenses a PCollection into a single list.

expand(pcoll)

class apache_beam.transforms.combiners.ToSet(has_defaults=True)

Bases: CombinerWithoutDefaults

A global CombineFn that condenses a PCollection into a set.

expand(pcoll)

class apache_beam.transforms.combiners.Latest

Bases: object

Combiners for computing the latest element

class Globally(has_defaults=True)

Bases: CombinerWithoutDefaults

Compute the element with the latest timestamp from a PCollection.

static add_timestamp(element, timestamp=TimestampParam)

expand(pcoll)

class PerKey(label: str | None = None)

Bases: PTransform

Compute elements with the latest timestamp for each key from a keyed PCollection

static add_timestamp(element, timestamp=TimestampParam)

expand(pcoll)

class apache_beam.transforms.combiners.CountCombineFn(*unused_args, **unused_kwargs)

Bases: CombineFn

CombineFn for computing PCollection size.

create_accumulator()

add_input(accumulator, element)

add_inputs(accumulator, elements)

merge_accumulators(accumulators)

extract_output(accumulator)

class apache_beam.transforms.combiners.MeanCombineFn(*unused_args, **unused_kwargs)

Bases: CombineFn

CombineFn for computing an arithmetic mean.

create_accumulator()

add_input(sum_count, element)

merge_accumulators(accumulators)

extract_output(sum_count)

for_input_type(input_type)

class apache_beam.transforms.combiners.SampleCombineFn(n)

Bases: CombineFn

CombineFn for all Sample transforms.

setup()

create_accumulator()

add_input(heap, element)

merge_accumulators(heaps)

compact(heap)

extract_output(heap)

teardown()

class apache_beam.transforms.combiners.TopCombineFn(n, key=None, reverse=False)

Bases: CombineFn

CombineFn doing the combining for all of the Top transforms.

This CombineFn uses a key or reverse operator to rank the elements.

Parameters:

• key – (optional) a mapping of elements to a comparable key, similar to the key argument of Python’s sorting methods.

• reverse – (optional) whether to order things smallest to largest, rather than largest to smallest

display_data()

create_accumulator(*args, **kwargs)

add_input(accumulator, element, *args, **kwargs)

merge_accumulators(accumulators, *args, **kwargs)

compact(accumulator, *args, **kwargs)

extract_output(accumulator, *args, **kwargs)

class apache_beam.transforms.combiners.ToDictCombineFn(*unused_args, **unused_kwargs)

Bases: CombineFn

CombineFn for to_dict.

create_accumulator()

add_input(accumulator, element)

merge_accumulators(accumulators)

extract_output(accumulator)

class apache_beam.transforms.combiners.ToListCombineFn(*unused_args, **unused_kwargs)

Bases: CombineFn

CombineFn for to_list.

create_accumulator()

add_input(accumulator, element)

merge_accumulators(accumulators)

extract_output(accumulator)

class apache_beam.transforms.combiners.ToSetCombineFn(*unused_args, **unused_kwargs)

Bases: CombineFn

CombineFn for ToSet.

create_accumulator()

add_input(accumulator, element)

merge_accumulators(accumulators)

extract_output(accumulator)

class apache_beam.transforms.combiners.LatestCombineFn(*unused_args, **unused_kwargs)

Bases: CombineFn

CombineFn to get the element with the latest timestamp from a PCollection.

create_accumulator()

add_input(accumulator, element)

merge_accumulators(accumulators)

extract_output(accumulator)