apache_beam.transforms.external module

Defines Transform whose expansion is implemented elsewhere.

No backward compatibility guarantees. Everything in this module is experimental.

apache_beam.transforms.external.iter_urns(coder, context=None)

class apache_beam.transforms.external.PayloadBuilder

Bases: object

Abstract base class for building payloads to pass to ExternalTransform.

build()

Returns:ExternalConfigurationPayload

payload()

The serialized ExternalConfigurationPayload

Returns:bytes

class apache_beam.transforms.external.SchemaBasedPayloadBuilder(values, schema)

Bases: apache_beam.transforms.external.PayloadBuilder

Base class for building payloads based on a schema that provides type information for each configuration value to encode.

Note that if the schema defines a type as Optional, the corresponding value will be omitted from the encoded payload, and thus the native transform will determine the default.

Parameters:

• values – mapping of config names to values
• schema – mapping of config names to types

build()

Returns:ExternalConfigurationPayload

class apache_beam.transforms.external.ImplicitSchemaPayloadBuilder(values)

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload that generates a schema from the provided values.

class apache_beam.transforms.external.NamedTupleBasedPayloadBuilder(tuple_instance)

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on a NamedTuple schema.

Parameters:tuple_instance – an instance of a typing.NamedTuple

class apache_beam.transforms.external.AnnotationBasedPayloadBuilder(transform, **values)

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on an external transform’s type annotations.

Supported in python 3 only.

Parameters:

• transform – a PTransform instance or class. type annotations will be gathered from its __init__ method
• values – values to encode

class apache_beam.transforms.external.DataclassBasedPayloadBuilder(transform)

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on an external transform that uses dataclasses.

Supported in python 3 only.

Parameters:transform – a dataclass-decorated PTransform instance from which to gather type annotations and values

class apache_beam.transforms.external.ExternalTransform(urn, payload, expansion_service=None)

Bases: apache_beam.transforms.ptransform.PTransform

External provides a cross-language transform via expansion services in foreign SDKs.

Experimental; no backwards compatibility guarantees.

Wrapper for an external transform with the given urn and payload.

Parameters:

• urn – the unique beam identifier for this transform
• payload – the payload, either as a byte string or a PayloadBuilder
• expansion_service – an expansion service implementing the beam ExpansionService protocol, either as an object with an Expand method or an address (as a str) to a grpc server that provides this method.

default_label()

classmethod get_local_namespace()

classmethod outer_namespace(namespace)

expand(pvalueish)

to_runner_api_transform(context, full_label)

class apache_beam.transforms.external.JavaJarExpansionService(path_to_jar, extra_args=None)

Bases: object

An expansion service based on an Java Jar file.

This can be passed into an ExternalTransform as the expansion_service argument which will spawn a subprocess using this jar to expand the transform.

Expand(request, context)

class apache_beam.transforms.external.BeamJarExpansionService(gradle_target, extra_args=None, gradle_appendix=None)

Bases: apache_beam.transforms.external.JavaJarExpansionService

An expansion service based on an Beam Java Jar file.

Attempts to use a locally-build copy of the jar based on the gradle target, if it exists, otherwise attempts to download it (with caching) from the apache maven repository.

apache_beam.transforms.external.memoize(func)