apache_beam.transforms.external module

Defines Transform whose expansion is implemented elsewhere.

No backward compatibility guarantees. Everything in this module is experimental.

apache_beam.transforms.external.convert_to_typing_type(type_)

apache_beam.transforms.external.iter_urns(coder, context=None)

class apache_beam.transforms.external.PayloadBuilder

Bases: object

Abstract base class for building payloads to pass to ExternalTransform.

build()

Returns:ExternalConfigurationPayload

payload()

The serialized ExternalConfigurationPayload

Returns:bytes

class apache_beam.transforms.external.SchemaBasedPayloadBuilder

Bases: apache_beam.transforms.external.PayloadBuilder

Base class for building payloads based on a schema that provides type information for each configuration value to encode.

build()

class apache_beam.transforms.external.ImplicitSchemaPayloadBuilder(values)

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload that generates a schema from the provided values.

class apache_beam.transforms.external.NamedTupleBasedPayloadBuilder(tuple_instance)

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on a NamedTuple schema.

Parameters:tuple_instance – an instance of a typing.NamedTuple

class apache_beam.transforms.external.JavaClassLookupPayloadBuilder(class_name)

Bases: apache_beam.transforms.external.PayloadBuilder

Builds a payload for directly instantiating a Java transform using a constructor and builder methods.

Parameters:class_name – fully qualified name of the transform class.

IGNORED_ARG_FORMAT = 'ignore%d'

build()

with_constructor(*args, **kwargs)

Specifies the Java constructor to use. Arguments provided using args and kwargs will be applied to the Java transform constructor in the specified order.

Parameters:

• args – parameter values of the constructor.
• kwargs – parameter names and values of the constructor.

with_constructor_method(method_name, *args, **kwargs)

Specifies the Java constructor method to use. Arguments provided using args and kwargs will be applied to the Java transform constructor method in the specified order.

Parameters:

• method_name – name of the constructor method.
• args – parameter values of the constructor method.
• kwargs – parameter names and values of the constructor method.

add_builder_method(method_name, *args, **kwargs)

Specifies a Java builder method to be invoked after instantiating the Java transform class. Specified builder method will be applied in order. Arguments provided using args and kwargs will be applied to the Java transform builder method in the specified order.

Parameters:

• method_name – name of the builder method.
• args – parameter values of the builder method.
• kwargs – parameter names and values of the builder method.

class apache_beam.transforms.external.JavaExternalTransform(class_name, expansion_service=None, classpath=None)

Bases: apache_beam.transforms.ptransform.PTransform

A proxy for Java-implemented external transforms.

One builds these transforms just as one would in Java, e.g.:

transform = JavaExternalTransform('fully.qualified.ClassName'
    )(contructorArg, ... ).builderMethod(...)

or:

JavaExternalTransform('fully.qualified.ClassName').staticConstructor(
    ...).builderMethod1(...).builderMethod2(...)

Parameters:

• class_name – fully qualified name of the java class
• expansion_service – (Optional) an expansion service to use. If none is provided, a default expansion service will be started.
• classpath – (Optional) A list paths to additional jars to place on the expansion service classpath.

expand(pcolls)

class apache_beam.transforms.external.AnnotationBasedPayloadBuilder(transform, **values)

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on an external transform’s type annotations.

Parameters:

• transform – a PTransform instance or class. type annotations will be gathered from its __init__ method
• values – values to encode

class apache_beam.transforms.external.DataclassBasedPayloadBuilder(transform)

Bases: apache_beam.transforms.external.SchemaBasedPayloadBuilder

Build a payload based on an external transform that uses dataclasses.

Parameters:transform – a dataclass-decorated PTransform instance from which to gather type annotations and values

class apache_beam.transforms.external.ExternalTransform(urn, payload, expansion_service=None)

Bases: apache_beam.transforms.ptransform.PTransform

External provides a cross-language transform via expansion services in foreign SDKs.

Experimental; no backwards compatibility guarantees.

Wrapper for an external transform with the given urn and payload.

Parameters:

• urn – the unique beam identifier for this transform
• payload – the payload, either as a byte string or a PayloadBuilder
• expansion_service – an expansion service implementing the beam ExpansionService protocol, either as an object with an Expand method or an address (as a str) to a grpc server that provides this method.

replace_named_inputs(named_inputs)

replace_named_outputs(named_outputs)

default_label()

classmethod get_local_namespace()

classmethod outer_namespace(namespace)

expand(pvalueish)

to_runner_api_transform(context, full_label)

class apache_beam.transforms.external.ExpansionAndArtifactRetrievalStub(channel, **kwargs)

Bases: apache_beam.portability.api.beam_expansion_api_pb2_grpc.ExpansionServiceStub

artifact_service()

class apache_beam.transforms.external.JavaJarExpansionService(path_to_jar, extra_args=None, classpath=None)

Bases: object

An expansion service based on an Java Jar file.

This can be passed into an ExternalTransform as the expansion_service argument which will spawn a subprocess using this jar to expand the transform.

class apache_beam.transforms.external.BeamJarExpansionService(gradle_target, extra_args=None, gradle_appendix=None, classpath=None)

Bases: apache_beam.transforms.external.JavaJarExpansionService

An expansion service based on an Beam Java Jar file.

Attempts to use a locally-built copy of the jar based on the gradle target, if it exists, otherwise attempts to download and cache the released artifact corresponding to this version of Beam from the apache maven repository.

apache_beam.transforms.external.memoize(func)