apache_beam.typehints package

Submodules

apache_beam.typehints.decorators module

Type hinting decorators allowing static or runtime type-checking for the SDK.

This module defines decorators which utilize the type-hints defined in ‘type_hints.py’ to allow annotation of the types of function arguments and return values.

Type-hints for functions are annotated using two separate decorators. One is for type-hinting the types of function arguments, the other for type-hinting the function return value. Type-hints can either be specified in the form of positional arguments:

@with_input_types(int, int)
def add(a, b):
  return a + b

Keyword arguments:

@with_input_types(a=int, b=int)
def add(a, b):
  return a + b

Or even a mix of both:

@with_input_types(int, b=int)
def add(a, b):
  return a + b

Example usage for type-hinting arguments only:

@with_input_types(s=str)
def to_lower(a):
  return a.lower()

Example usage for type-hinting return values only:

@with_output_types(Tuple[int, bool])
def compress_point(ec_point):
  return ec_point.x, ec_point.y < 0

Example usage for type-hinting both arguments and return values:

@with_input_types(a=int)
@with_output_types(str)
def int_to_str(a):
  return str(a)

Type-hinting a function with arguments that unpack tuples are also supported. As an example, such a function would be defined as:

def foo((a, b)):
  ...

The valid type-hint for such as function looks like the following:

@with_input_types(a=int, b=int)
def foo((a, b)):
  ...

Notice that we hint the type of each unpacked argument independently, rather than hinting the type of the tuple as a whole (Tuple[int, int]).

Optionally, type-hints can be type-checked at runtime. To toggle this behavior this module defines two functions: ‘enable_run_time_type_checking’ and ‘disable_run_time_type_checking’. NOTE: for this toggle behavior to work properly it must appear at the top of the module where all functions are defined, or before importing a module containing type-hinted functions.

apache_beam.typehints.decorators.with_input_types(*positional_hints, **keyword_hints)

A decorator that type-checks defined type-hints with passed func arguments.

All type-hinted arguments can be specified using positional arguments, keyword arguments, or a mix of both. Additionaly, all function arguments must be type-hinted in totality if even one parameter is type-hinted.

Once fully decorated, if the arguments passed to the resulting function violate the type-hint constraints defined, a TypeCheckError detailing the error will be raised.

To be used as:

* @with_input_types(s=str)  # just @with_input_types(str) will work too.
  def upper(s):
    return s.upper()

Or:

* @with_input_types(ls=List[Tuple[int, int])
  def increment(ls):
    [(i + 1, j + 1) for (i,j) in ls]

Parameters:

• *positional_hints – Positional type-hints having identical order as the function’s formal arguments. Values for this argument must either be a built-in Python type or an instance of a TypeContraint created by ‘indexing’ a CompositeTypeHint instance with a type parameter.
• **keyword_hints – Keyword arguments mirroring the names of the parameters to the decorated functions. The value of each keyword argument must either be one of the allowed built-in Python types, a custom class, or an instance of a TypeContraint created by ‘indexing’ a CompositeTypeHint instance with a type parameter.

Raises:

• ValueError – If not all function arguments have corresponding type-hints specified. Or if the inner wrapper function isn’t passed a function object.
• TypeCheckError – If the any of the passed type-hint constraints are not a type or TypeContraint instance.

Returns:

The original function decorated such that it enforces type-hint constraints for all received function arguments.

apache_beam.typehints.decorators.with_output_types(*return_type_hint, **kwargs)

A decorator that type-checks defined type-hints for return values(s).

This decorator will type-check the return value(s) of the decorated function.

Only a single type-hint is accepted to specify the return type of the return value. If the function to be decorated has multiple return values, then one should use: ‘Tuple[type_1, type_2]’ to annotate the types of the return values.

If the ultimate return value for the function violates the specified type-hint a TypeCheckError will be raised detailing the type-constraint violation.

This decorator is intended to be used like:

* @with_output_types(Set[Coordinate])
  def parse_ints(ints):
    ....
    return [Coordinate.from_int(i) for i in ints]

Or with a simple type-hint:

* @with_output_types(bool)
  def negate(p):
    return not p if p else p

Parameters:

• *return_type_hint – A type-hint specifying the proper return type of the function. This argument should either be a built-in Python type or an instance of a ‘TypeConstraint’ created by ‘indexing’ a ‘CompositeTypeHint’.
• **kwargs – Not used.

Raises:

• ValueError – If any kwarg parameters are passed in, or the length of ‘return_type_hint’ is greater than 1. Or if the inner wrapper function isn’t passed a function object.
• TypeCheckError – If the ‘return_type_hint’ object is in invalid type-hint.

Returns:

The original function decorated such that it enforces type-hint constraints for all return values.

class apache_beam.typehints.decorators.WithTypeHints(*unused_args, **unused_kwargs)

Bases: object

A mixin class that provides the ability to set and retrieve type hints.

default_type_hints()

get_type_hints()

with_input_types(*arg_hints, **kwarg_hints)

with_output_types(*arg_hints, **kwarg_hints)

exception apache_beam.typehints.decorators.TypeCheckError

Bases: exceptions.Exception

apache_beam.typehints.opcodes module

Defines the actions various bytecodes have on the frame.

Each function here corresponds to a bytecode documented in https://docs.python.org/2/library/dis.html. The first argument is a (mutable) FrameState object, the second the integer opcode argument.

Bytecodes with more complicated behavior (e.g. modifying the program counter) are handled inline rather than here.

For internal use only; no backwards-compatibility guarantees.

apache_beam.typehints.opcodes.binary_add(state, unused_arg)

apache_beam.typehints.opcodes.binary_and(state, unused_arg)

apache_beam.typehints.opcodes.binary_divide(state, unused_arg)

apache_beam.typehints.opcodes.binary_floor_divide(state, unused_arg)

apache_beam.typehints.opcodes.binary_lshift(state, unused_arg)

apache_beam.typehints.opcodes.binary_modulo(state, unused_arg)

apache_beam.typehints.opcodes.binary_multiply(state, unused_arg)

apache_beam.typehints.opcodes.binary_or(state, unused_arg)

apache_beam.typehints.opcodes.binary_power(state, unused_arg)

apache_beam.typehints.opcodes.binary_rshift(state, unused_arg)

apache_beam.typehints.opcodes.binary_subscr(state, unused_arg)

apache_beam.typehints.opcodes.binary_subtract(state, unused_arg)

apache_beam.typehints.opcodes.binary_true_divide(state, unused_arg)

apache_beam.typehints.opcodes.binary_xor(state, unused_arg)

apache_beam.typehints.opcodes.build_class(state, unused_arg)

apache_beam.typehints.opcodes.build_list(state, arg)

apache_beam.typehints.opcodes.build_map(state, unused_arg)

apache_beam.typehints.opcodes.build_slice(state, arg)

apache_beam.typehints.opcodes.build_tuple(state, arg)

apache_beam.typehints.opcodes.call_function(state, arg, has_var=False, has_kw=False)

apache_beam.typehints.opcodes.call_function_kw(state, arg)

apache_beam.typehints.opcodes.call_function_var(state, arg)

apache_beam.typehints.opcodes.call_function_var_wk(state, arg)

apache_beam.typehints.opcodes.compare_op(state, unused_arg)

apache_beam.typehints.opcodes.delete_attr(unused_state, unused_arg)

apache_beam.typehints.opcodes.delete_fast(state, arg)

apache_beam.typehints.opcodes.delete_global(unused_state, unused_arg)

apache_beam.typehints.opcodes.delete_slice_0(unused_state, unused_arg)

apache_beam.typehints.opcodes.delete_slice_1(unused_state, unused_arg)

apache_beam.typehints.opcodes.delete_slice_2(unused_state, unused_arg)

apache_beam.typehints.opcodes.delete_slice_3(unused_state, unused_arg)

apache_beam.typehints.opcodes.dup_top(state, unused_arg)

apache_beam.typehints.opcodes.dup_topx(state, arg)

apache_beam.typehints.opcodes.exec_stmt(state, unused_arg)

apache_beam.typehints.opcodes.get_iter(state, unused_arg)

apache_beam.typehints.opcodes.import_from(state, unused_arg)

apache_beam.typehints.opcodes.import_name(state, unused_arg)

apache_beam.typehints.opcodes.inpalce_or(state, unused_arg)

apache_beam.typehints.opcodes.inplace_add(state, unused_arg)

apache_beam.typehints.opcodes.inplace_and(state, unused_arg)

apache_beam.typehints.opcodes.inplace_divide(state, unused_arg)

apache_beam.typehints.opcodes.inplace_floor_divide(state, unused_arg)

apache_beam.typehints.opcodes.inplace_lshift(state, unused_arg)

apache_beam.typehints.opcodes.inplace_modulo(state, unused_arg)

apache_beam.typehints.opcodes.inplace_multiply(state, unused_arg)

apache_beam.typehints.opcodes.inplace_power(state, unused_arg)

apache_beam.typehints.opcodes.inplace_rshift(state, unused_arg)

apache_beam.typehints.opcodes.inplace_subtract(state, unused_arg)

apache_beam.typehints.opcodes.inplace_true_divide(state, unused_arg)

apache_beam.typehints.opcodes.inplace_xor(state, unused_arg)

apache_beam.typehints.opcodes.list_append(state, arg)

apache_beam.typehints.opcodes.load_attr(state, arg)

apache_beam.typehints.opcodes.load_closure(state, unused_arg)

apache_beam.typehints.opcodes.load_const(state, arg)

apache_beam.typehints.opcodes.load_deref(state, arg)

apache_beam.typehints.opcodes.load_fast(state, arg)

apache_beam.typehints.opcodes.load_global(state, arg)

apache_beam.typehints.opcodes.load_locals(state, unused_arg)

apache_beam.typehints.opcodes.load_name(state, unused_arg)

apache_beam.typehints.opcodes.make_closure(state, arg)

apache_beam.typehints.opcodes.make_function(state, arg)

apache_beam.typehints.opcodes.nop(unused_state, unused_arg)

apache_beam.typehints.opcodes.pop_one(state, unused_arg)

apache_beam.typehints.opcodes.pop_three(state, unused_arg)

apache_beam.typehints.opcodes.pop_top(state, unused_arg)

apache_beam.typehints.opcodes.pop_two(state, unused_arg)

apache_beam.typehints.opcodes.print_item(state, unused_arg)

apache_beam.typehints.opcodes.print_newline(unused_state, unused_arg)

apache_beam.typehints.opcodes.push_value(v)

apache_beam.typehints.opcodes.rot_four(state, unused_arg)

apache_beam.typehints.opcodes.rot_n(state, n)

apache_beam.typehints.opcodes.rot_three(state, unused_arg)

apache_beam.typehints.opcodes.rot_two(state, unused_arg)

apache_beam.typehints.opcodes.slice_0(unused_state, unused_arg)

apache_beam.typehints.opcodes.slice_1(state, unused_arg)

apache_beam.typehints.opcodes.slice_2(state, unused_arg)

apache_beam.typehints.opcodes.slice_3(state, unused_arg)

apache_beam.typehints.opcodes.store_attr(state, unused_arg)

apache_beam.typehints.opcodes.store_fast(state, arg)

apache_beam.typehints.opcodes.store_global(state, unused_arg)

apache_beam.typehints.opcodes.store_map(state, unused_arg)

apache_beam.typehints.opcodes.store_slice_0(unused_state, unused_arg)

apache_beam.typehints.opcodes.store_slice_1(unused_state, unused_arg)

apache_beam.typehints.opcodes.store_slice_2(unused_state, unused_arg)

apache_beam.typehints.opcodes.store_slice_3(unused_state, unused_arg)

apache_beam.typehints.opcodes.store_subscr(unused_state, unused_args)

apache_beam.typehints.opcodes.symmetric_binary_op(state, unused_arg)

apache_beam.typehints.opcodes.unary(state, unused_arg)

apache_beam.typehints.opcodes.unary_convert(state, unused_arg)

apache_beam.typehints.opcodes.unary_invert(state, unused_arg)

apache_beam.typehints.opcodes.unary_negative(state, unused_arg)

apache_beam.typehints.opcodes.unary_not(state, unused_arg)

apache_beam.typehints.opcodes.unary_positive(state, unused_arg)

apache_beam.typehints.opcodes.unpack_sequence(state, arg)

apache_beam.typehints.trivial_inference module

Trivial type inference for simple functions.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.typehints.trivial_inference.BoundMethod(unbound)

Bases: object

Used to create a bound method when we only know the type of the instance.

class apache_beam.typehints.trivial_inference.Const(value)

Bases: object

static unwrap(x)

static unwrap_all(xs)

class apache_beam.typehints.trivial_inference.FrameState(f, local_vars=None, stack=())

Bases: object

Stores the state of the frame at a particular point of execution.

closure_type(i)

const_type(i)

copy()

get_global(i)

get_name(i)

exception apache_beam.typehints.trivial_inference.TypeInferenceError

Bases: exceptions.ValueError

Error to raise when type inference failed.

apache_beam.typehints.trivial_inference.element_type(hint)

Returns the element type of a composite type.

apache_beam.typehints.trivial_inference.hashable(c)

apache_beam.typehints.trivial_inference.infer_return_type(c, input_types, debug=False, depth=5)

Analyses a callable to deduce its return type.

Parameters:

• c – A Python callable to infer the return type of.
• input_types – A sequence of inputs corresponding to the input types.
• debug – Whether to print verbose debugging information.
• depth – Maximum inspection depth during type inference.

Returns:

A TypeConstraint that that the return value of this function will (likely) satisfy given the specified inputs.

apache_beam.typehints.trivial_inference.infer_return_type_func(f, input_types, debug=False, depth=0)

Analyses a function to deduce its return type.

Parameters:

• f – A Python function object to infer the return type of.
• input_types – A sequence of inputs corresponding to the input types.
• debug – Whether to print verbose debugging information.
• depth – Maximum inspection depth during type inference.

Returns:

A TypeConstraint that that the return value of this function will (likely) satisfy given the specified inputs.

Raises:

TypeInferenceError – if no type can be inferred.

apache_beam.typehints.trivial_inference.instance_to_type(o)

Given a Python object o, return the corresponding type hint.

apache_beam.typehints.trivial_inference.key_value_types(kv_type)

Returns the key and value type of a KV type.

apache_beam.typehints.trivial_inference.union(a, b)

Returns the union of two types or Const values.

apache_beam.typehints.trivial_inference.union_list(xs, ys)

apache_beam.typehints.typecheck module

Runtime type checking support.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.typehints.typecheck.AbstractDoFnWrapper(dofn)

Bases: apache_beam.transforms.core.DoFn

An abstract class to create wrapper around DoFn

finish_bundle(*args, **kwargs)

is_process_bounded()

process(*args, **kwargs)

start_bundle(*args, **kwargs)

wrapper(method, args, kwargs)

class apache_beam.typehints.typecheck.OutputCheckWrapperDoFn(dofn, full_label)

Bases: apache_beam.typehints.typecheck.AbstractDoFnWrapper

A DoFn that verifies against common errors in the output type.

wrapper(method, args, kwargs)

class apache_beam.typehints.typecheck.TypeCheckWrapperDoFn(dofn, type_hints, label=None)

Bases: apache_beam.typehints.typecheck.AbstractDoFnWrapper

A wrapper around a DoFn which performs type-checking of input and output.

process(*args, **kwargs)

wrapper(method, args, kwargs)

apache_beam.typehints.typehints module

Syntax & semantics for type-hinting custom-functions/PTransforms in the SDK.

This module defines type-hinting objects and the corresponding syntax for type-hinting function arguments, function return types, or PTransform object themselves. TypeHint’s defined in the module can be used to implement either static or run-time type-checking in regular Python code.

Type-hints are defined by ‘indexing’ a type-parameter into a defined CompositeTypeHint instance:

• ‘List[int]’.

Valid type-hints are partitioned into two categories: simple, and composite.

Simple type hints are type hints based on a subset of Python primitive types: int, bool, float, str, object, None, and bytes. No other primitive types are allowed.

Composite type-hints are reserved for hinting the types of container-like Python objects such as ‘list’. Composite type-hints can be parameterized by an inner simple or composite type-hint, using the ‘indexing’ syntax. In order to avoid conflicting with the namespace of the built-in container types, when specifying this category of type-hints, the first letter should capitalized. The following composite type-hints are permitted. NOTE: ‘T’ can be any of the type-hints listed or a simple Python type:

• Any
• Union[T, T, T]
• Optional[T]
• Tuple[T, T]
• Tuple[T, ...]
• List[T]
• KV[T, T]
• Dict[T, T]
• Set[T]
• Iterable[T]
• Iterator[T]
• Generator[T]

Type-hints can be nested, allowing one to define type-hints for complex types:

• ‘List[Tuple[int, int, str]]

In addition, type-hints can be used to implement run-time type-checking via the ‘type_check’ method on each TypeConstraint.

apache_beam.typehints.typehints.WindowedValue

alias of WindowedTypeConstraint

class apache_beam.typehints.typehints.TypeVariable(name)

Bases: apache_beam.typehints.typehints.AnyTypeConstraint

bind_type_variables(bindings)

match_type_variables(concrete_type)

Module contents

A package defining the syntax and decorator semantics for type-hints.