apache_beam.typehints.batch module

Utilities for working with batched types in the Beam SDK.

A batched type is a type B that is logically equivalent to Sequence[E], where E is some other type. Typically B has a different physical representation than Sequence[E] for performance reasons.

A trivial example is B=np.array(dtype=np.int64), E=int.

class apache_beam.typehints.batch.BatchConverter(batch_type, element_type)

Bases: typing.Generic

produce_batch(elements: Sequence[E]) → B

Convert an instance of List[E] to a single instance of B.

explode_batch(batch: B) → Iterator[E]

Convert an instance of B to Iterator[E].

combine_batches(batches: Sequence[B]) → B

get_length(batch: B) → int

estimate_byte_size(batch)

static register(batch_converter_constructor: Callable[[type, type], BatchConverter])

static from_typehints(*, element_type, batch_type) → apache_beam.typehints.batch.BatchConverter

batch_type

element_type