apache_beam.transforms.external module

Defines Transform whose expansion is implemented elsewhere.

No backward compatibility guarantees. Everything in this module is experimental.

apache_beam.transforms.external.iter_urns(coder, context=None)[source]
class apache_beam.transforms.external.PayloadBuilder[source]

Bases: object

Abstract base class for building payloads to pass to ExternalTransform.

build()[source]
Returns:ExternalConfigurationPayload
payload()[source]

The serialized ExternalConfigurationPayload

Returns:bytes
class apache_beam.transforms.external.SchemaBasedPayloadBuilder(values, schema)[source]

Bases: apache_beam.transforms.external.PayloadBuilder

Base class for building payloads based on a schema that provides type information for each configuration value to encode.

Note that if the schema defines a type as Optional, the corresponding value will be omitted from the encoded payload, and thus the native transform will determine the default.

Parameters:
  • values – mapping of config names to values
  • schema – mapping of config names to types
build()[source]
Returns:ExternalConfigurationPayload
class apache_beam.transforms.external.ImplicitSchemaPayloadBuilder(values)[source]

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload that generates a schema from the provided values.

class apache_beam.transforms.external.NamedTupleBasedPayloadBuilder(tuple_instance)[source]

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on a NamedTuple schema.

Parameters:tuple_instance – an instance of a typing.NamedTuple
class apache_beam.transforms.external.AnnotationBasedPayloadBuilder(transform, **values)[source]

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on an external transform’s type annotations.

Supported in python 3 only.

Parameters:
  • transform – a PTransform instance or class. type annotations will be gathered from its __init__ method
  • values – values to encode
class apache_beam.transforms.external.DataclassBasedPayloadBuilder(transform)[source]

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on an external transform that uses dataclasses.

Supported in python 3 only.

Parameters:transform – a dataclass-decorated PTransform instance from which to gather type annotations and values
class apache_beam.transforms.external.ExternalTransform(urn, payload, expansion_service=None)[source]

Bases: apache_beam.transforms.ptransform.PTransform

External provides a cross-language transform via expansion services in foreign SDKs.

Experimental; no backwards compatibility guarantees.

Wrapper for an external transform with the given urn and payload.

Parameters:
  • urn – the unique beam identifier for this transform
  • payload – the payload, either as a byte string or a PayloadBuilder
  • expansion_service – an expansion service implementing the beam ExpansionService protocol, either as an object with an Expand method or an address (as a str) to a grpc server that provides this method.
default_label()[source]
classmethod get_local_namespace()[source]
classmethod outer_namespace(namespace)[source]
expand(pvalueish)[source]
to_runner_api_transform(context, full_label)[source]
class apache_beam.transforms.external.JavaJarExpansionService(path_to_jar, extra_args=None)[source]

Bases: object

An expansion service based on an Java Jar file.

This can be passed into an ExternalTransform as the expansion_service argument which will spawn a subprocess using this jar to expand the transform.

Expand(request, context)[source]
class apache_beam.transforms.external.BeamJarExpansionService(gradle_target, extra_args=None, gradle_appendix=None)[source]

Bases: apache_beam.transforms.external.JavaJarExpansionService

An expansion service based on an Beam Java Jar file.

Attempts to use a locally-build copy of the jar based on the gradle target, if it exists, otherwise attempts to download it (with caching) from the apache maven repository.

apache_beam.transforms.external.memoize(func)[source]