apache_beam.typehints.batch module

Utilities for working with batched types in the Beam SDK.

A batched type is a type B that is logically equivalent to Sequence[E], where E is some other type. Typically B has a different physical representation than Sequence[E] for performance reasons.

A trivial example is B=np.array(dtype=np.int64), E=int.

class apache_beam.typehints.batch.BatchConverter(batch_type, element_type)[source]

Bases: typing.Generic

produce_batch(elements: Sequence[E]) → B[source]

Convert an instance of List[E] to a single instance of B.

explode_batch(batch: B) → Iterator[E][source]

Convert an instance of B to Iterator[E].

combine_batches(batches: Sequence[B]) → B[source]
get_length(batch: B) → int[source]
estimate_byte_size(batch)[source]
static register(batch_converter_constructor: Callable[[type, type], BatchConverter])[source]
static from_typehints(*, element_type, batch_type) → apache_beam.typehints.batch.BatchConverter[source]
batch_type
element_type