apache_beam.typehints.schemas module

Support for mapping python types to proto Schemas and back again.

Imposes a mapping between common Python types and Beam portable schemas (https://s.apache.org/beam-schemas):

Python              Schema
np.int8     <-----> BYTE
np.int16    <-----> INT16
np.int32    <-----> INT32
np.int64    <-----> INT64
int         ------> INT64
np.float32  <-----> FLOAT
np.float64  <-----> DOUBLE
float       ------> DOUBLE
bool        <-----> BOOLEAN
str         <-----> STRING
bytes       <-----> BYTES
ByteString  ------> BYTES
Timestamp   <-----> LogicalType(urn="beam:logical_type:micros_instant:v1")
Decimal     <-----> LogicalType(urn="beam:logical_type:fixed_decimal:v1")
Mapping     <-----> MapType
Sequence    <-----> ArrayType
NamedTuple  <-----> RowType
beam.Row    ------> RowType

One direction mapping of Python types from Beam portable schemas:

bytes

<—— LogicalType(urn=”beam:logical_type:fixed_bytes:v1”) <—— LogicalType(urn=”beam:logical_type:var_bytes:v1”)

str

<—— LogicalType(urn=”beam:logical_type:fixed_char:v1”) <—— LogicalType(urn=”beam:logical_type:var_char:v1”)

Timestamp

<—— LogicalType(urn=”beam:logical_type:millis_instant:v1”)

Note that some of these mappings are provided as conveniences, but they are lossy and will not survive a roundtrip from python to Beam schemas and back. For example, the Python type int will map to INT64 in Beam schemas but converting that back to a Python type will yield np.int64.

nullable=True on a Beam FieldType is represented in Python by wrapping the type in Optional.

This module is intended for internal use only. Nothing defined here provides any backwards-compatibility guarantee.

apache_beam.typehints.schemas.named_fields_to_schema(names_and_types: Union[Dict[str, type], Sequence[Tuple[str, type]]], schema_id: Optional[str] = None, schema_options: Optional[Sequence[Tuple[str, Any]]] = None, field_options: Optional[Dict[str, Sequence[Tuple[str, Any]]]] = None, schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>)

apache_beam.typehints.schemas.named_fields_from_schema(schema)

apache_beam.typehints.schemas.typing_to_runner_api(type_: type, schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>) → org.apache.beam.model.pipeline.v1.schema_pb2.FieldType

apache_beam.typehints.schemas.typing_from_runner_api(fieldtype_proto: org.apache.beam.model.pipeline.v1.schema_pb2.FieldType, schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>) → type

apache_beam.typehints.schemas.value_to_runner_api(type_proto: org.apache.beam.model.pipeline.v1.schema_pb2.FieldType, value, schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>) → org.apache.beam.model.pipeline.v1.schema_pb2.FieldValue

apache_beam.typehints.schemas.value_from_runner_api(type_proto: org.apache.beam.model.pipeline.v1.schema_pb2.FieldType, value_proto: org.apache.beam.model.pipeline.v1.schema_pb2.FieldValue, schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>) → org.apache.beam.model.pipeline.v1.schema_pb2.FieldValue

apache_beam.typehints.schemas.option_to_runner_api(option: Tuple[str, Any], schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>) → org.apache.beam.model.pipeline.v1.schema_pb2.Option

apache_beam.typehints.schemas.option_from_runner_api(option_proto: org.apache.beam.model.pipeline.v1.schema_pb2.Option, schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>) → Tuple[str, Any]

class apache_beam.typehints.schemas.SchemaTranslation(schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>)

Bases: object

typing_to_runner_api(type_: type) → org.apache.beam.model.pipeline.v1.schema_pb2.FieldType

atomic_value_from_runner_api(atomic_type: <google.protobuf.internal.enum_type_wrapper.EnumTypeWrapper object at 0x7fd74b6a83d0>, atomic_value: org.apache.beam.model.pipeline.v1.schema_pb2.AtomicTypeValue)

atomic_value_to_runner_api(atomic_type: <google.protobuf.internal.enum_type_wrapper.EnumTypeWrapper object at 0x7fd74b6a83d0>, value) → org.apache.beam.model.pipeline.v1.schema_pb2.AtomicTypeValue

value_from_runner_api(type_proto: org.apache.beam.model.pipeline.v1.schema_pb2.FieldType, value_proto: org.apache.beam.model.pipeline.v1.schema_pb2.FieldValue)

value_to_runner_api(typing_proto: org.apache.beam.model.pipeline.v1.schema_pb2.FieldType, value)

option_from_runner_api(option_proto: org.apache.beam.model.pipeline.v1.schema_pb2.Option) → Tuple[str, Any]

option_to_runner_api(option: Tuple[str, Any]) → org.apache.beam.model.pipeline.v1.schema_pb2.Option

typing_from_runner_api(fieldtype_proto: org.apache.beam.model.pipeline.v1.schema_pb2.FieldType) → type

named_tuple_from_schema(schema: org.apache.beam.model.pipeline.v1.schema_pb2.Schema) → type

apache_beam.typehints.schemas.named_tuple_from_schema(schema, schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>) → type

apache_beam.typehints.schemas.named_tuple_to_schema(named_tuple, schema_registry: apache_beam.typehints.schema_registry.SchemaTypeRegistry = <apache_beam.typehints.schema_registry.SchemaTypeRegistry object>) → org.apache.beam.model.pipeline.v1.schema_pb2.Schema

apache_beam.typehints.schemas.schema_from_element_type(element_type: type) → org.apache.beam.model.pipeline.v1.schema_pb2.Schema

Get a schema for the given PCollection element_type.

Returns schema as a list of (name, python_type) tuples

apache_beam.typehints.schemas.named_fields_from_element_type(element_type: type) → List[Tuple[str, type]]

apache_beam.typehints.schemas.union_schema_type(element_types)

Returns a schema whose fields are the union of each corresponding field.

element_types must be a set of schema-aware types whose fields have the same naming and ordering.

class apache_beam.typehints.schemas.LogicalTypeRegistry

Bases: object

add(urn, logical_type)

get_logical_type_by_urn(urn)

get_urn_by_logial_type(logical_type)

get_logical_type_by_language_type(representation_type)

class apache_beam.typehints.schemas.LogicalType

Bases: typing.Generic

classmethod urn()

Return the URN used to identify this logical type

classmethod language_type()

Return the language type this LogicalType encodes.

The returned type should match LanguageT

classmethod representation_type()

Return the type of the representation this LogicalType uses to encode the language type.

The returned type should match RepresentationT

classmethod argument_type()

Return the type of the argument used for variations of this LogicalType.

The returned type should match ArgT

argument()

Return the argument for this instance of the LogicalType.

to_representation_type(value)

Convert an instance of LanguageT to RepresentationT.

to_language_type(value)

Convert an instance of RepresentationT to LanguageT.

classmethod register_logical_type(logical_type_cls)

Register an implementation of LogicalType.

classmethod from_typing(typ)

Construct an instance of a registered LogicalType implementation given a typing.

Raises ValueError if no registered LogicalType implementation can encode the given typing.

classmethod from_runner_api(logical_type_proto)

Construct an instance of a registered LogicalType implementation given a proto LogicalType.

Raises ValueError if no LogicalType registered for the given URN.

class apache_beam.typehints.schemas.NoArgumentLogicalType

Bases: apache_beam.typehints.schemas.LogicalType

classmethod argument_type()

argument()

class apache_beam.typehints.schemas.PassThroughLogicalType

Bases: apache_beam.typehints.schemas.LogicalType

A base class for LogicalTypes that use the same type as the underlying representation type.

to_language_type(value)

classmethod representation_type()

to_representation_type(value)

class apache_beam.typehints.schemas.MicrosInstantRepresentation(seconds, micros)

Bases: tuple

Create new instance of MicrosInstantRepresentation(seconds, micros)

micros

Alias for field number 1

seconds

Alias for field number 0

class apache_beam.typehints.schemas.MillisInstant

Bases: apache_beam.typehints.schemas.NoArgumentLogicalType

Millisecond-precision instant logical type handles values consistent with that encoded by InstantCoder in the Java SDK.

This class handles apache_beam.utils.timestamp.Timestamp language type as MicrosInstant, but it only provides millisecond precision, because it is aimed to handle data encoded by Java sdk’s InstantCoder which has same precision level.

Timestamp is handled by MicrosInstant by default. In some scenario, such as read from cross-language transform with rows containing InstantCoder encoded timestamps, one may need to override the mapping of Timetamp to MillisInstant. To do this, re-register this class with register_logical_type().

classmethod representation_type()

classmethod urn()

classmethod language_type()

to_language_type(value)

class apache_beam.typehints.schemas.MicrosInstant

Bases: apache_beam.typehints.schemas.NoArgumentLogicalType

Microsecond-precision instant logical type that handles Timestamp.

classmethod urn()

classmethod representation_type()

classmethod language_type()

to_representation_type(value)

to_language_type(value)

class apache_beam.typehints.schemas.PythonCallable

Bases: apache_beam.typehints.schemas.NoArgumentLogicalType

A logical type for PythonCallableSource objects.

classmethod urn()

classmethod representation_type()

classmethod language_type()

to_representation_type(value)

to_language_type(value)

class apache_beam.typehints.schemas.FixedPrecisionDecimalArgumentRepresentation(precision, scale)

Bases: tuple

Create new instance of FixedPrecisionDecimalArgumentRepresentation(precision, scale)

precision

Alias for field number 0

scale

Alias for field number 1

class apache_beam.typehints.schemas.DecimalLogicalType

Bases: apache_beam.typehints.schemas.NoArgumentLogicalType

A logical type for decimal objects handling values consistent with that encoded by BigDecimalCoder in the Java SDK.

classmethod urn()

classmethod representation_type()

classmethod language_type()

to_representation_type(value)

to_language_type(value)

class apache_beam.typehints.schemas.FixedPrecisionDecimalLogicalType(precision=-1, scale=0)

Bases: apache_beam.typehints.schemas.LogicalType

A wrapper of DecimalLogicalType that contains the precision value.

classmethod urn()

classmethod representation_type()

classmethod language_type()

to_representation_type(value)

to_language_type(value)

classmethod argument_type()

argument()

class apache_beam.typehints.schemas.FixedBytes(length: numpy.int32)

Bases: apache_beam.typehints.schemas.PassThroughLogicalType

A logical type for fixed-length bytes.

classmethod urn()

classmethod language_type() → type

to_language_type(value: bytes)

classmethod argument_type()

argument()

class apache_beam.typehints.schemas.VariableBytes(max_length: numpy.int32 = 2147483647)

Bases: apache_beam.typehints.schemas.PassThroughLogicalType

A logical type for variable-length bytes with specified maximum length.

classmethod urn()

classmethod language_type() → type

to_language_type(value: bytes)

classmethod argument_type()

argument()

class apache_beam.typehints.schemas.FixedString(length: numpy.int32)

Bases: apache_beam.typehints.schemas.PassThroughLogicalType

A logical type for fixed-length string.

classmethod urn()

classmethod language_type() → type

to_language_type(value: str)

classmethod argument_type()

argument()

class apache_beam.typehints.schemas.VariableString(max_length: numpy.int32 = 2147483647)

Bases: apache_beam.typehints.schemas.PassThroughLogicalType

A logical type for variable-length string with specified maximum length.

classmethod urn()

classmethod language_type() → type

to_language_type(value: str)

classmethod argument_type()

argument()