apache_beam.typehints.schemas module

Support for mapping python types to proto Schemas and back again.

Imposes a mapping between common Python types and Beam portable schemas (https://s.apache.org/beam-schemas):

Python              Schema
np.int8     <-----> BYTE
np.int16    <-----> INT16
np.int32    <-----> INT32
np.int64    <-----> INT64
int         ------> INT64
np.float32  <-----> FLOAT
np.float64  <-----> DOUBLE
float       ------> DOUBLE
bool        <-----> BOOLEAN
str         <-----> STRING
bytes       <-----> BYTES
ByteString  ------> BYTES
Timestamp   <-----> LogicalType(urn="beam:logical_type:micros_instant:v1")
Mapping     <-----> MapType
Sequence    <-----> ArrayType
NamedTuple  <-----> RowType
beam.Row    ------> RowType

Note that some of these mappings are provided as conveniences, but they are lossy and will not survive a roundtrip from python to Beam schemas and back. For example, the Python type int will map to INT64 in Beam schemas but converting that back to a Python type will yield np.int64.

nullable=True on a Beam FieldType is represented in Python by wrapping the type in Optional.

This module is intended for internal use only. Nothing defined here provides any backwards-compatibility guarantee.

class apache_beam.typehints.schemas.SchemaTypeRegistry

Bases: object

add(typing, schema)

get_typing_by_id(unique_id)

get_schema_by_id(unique_id)

apache_beam.typehints.schemas.named_fields_to_schema(names_and_types)

apache_beam.typehints.schemas.named_fields_from_schema(schema)

apache_beam.typehints.schemas.typing_to_runner_api(type_)

apache_beam.typehints.schemas.typing_from_runner_api(fieldtype_proto)

apache_beam.typehints.schemas.named_tuple_from_schema(schema)

apache_beam.typehints.schemas.named_tuple_to_schema(named_tuple)

apache_beam.typehints.schemas.schema_from_element_type(element_type: type) → schema_pb2.Schema

Get a schema for the given PCollection element_type.

Returns schema as a list of (name, python_type) tuples

apache_beam.typehints.schemas.named_fields_from_element_type(element_type: type) → List[Tuple[str, type]]

class apache_beam.typehints.schemas.LogicalTypeRegistry

Bases: object

add(urn, logical_type)

get_logical_type_by_urn(urn)

get_urn_by_logial_type(logical_type)

get_logical_type_by_language_type(representation_type)

class apache_beam.typehints.schemas.LogicalType

Bases: typing.Generic

classmethod urn()

Return the URN used to identify this logical type

classmethod language_type()

Return the language type this LogicalType encodes.

The returned type should match LanguageT

classmethod representation_type()

Return the type of the representation this LogicalType uses to encode the language type.

The returned type should match RepresentationT

classmethod argument_type()

Return the type of the argument used for variations of this LogicalType.

The returned type should match ArgT

argument()

Return the argument for this instance of the LogicalType.

to_representation_type()

Convert an instance of LanguageT to RepresentationT.

to_language_type()

Convert an instance of RepresentationT to LanguageT.

classmethod register_logical_type(logical_type_cls)

Register an implementation of LogicalType.

classmethod from_typing(typ)

Construct an instance of a registered LogicalType implementation given a typing.

Raises ValueError if no registered LogicalType implementation can encode the given typing.

classmethod from_runner_api(logical_type_proto)

Construct an instance of a registered LogicalType implementation given a proto LogicalType.

Raises ValueError if no LogicalType registered for the given URN.

class apache_beam.typehints.schemas.NoArgumentLogicalType

Bases: apache_beam.typehints.schemas.LogicalType

classmethod argument_type()

argument()

class apache_beam.typehints.schemas.MicrosInstantRepresentation(seconds, micros)

Bases: tuple

Create new instance of MicrosInstantRepresentation(seconds, micros)

micros

Alias for field number 1

seconds

Alias for field number 0