apache_beam.yaml.yaml_provider module

This module defines Providers usable from yaml, which is a specification for where to find and how to invoke services that vend implementations of various PTransforms.

class apache_beam.yaml.yaml_provider.Provider

Bases: object

Maps transform types names and args to concrete PTransform instances.

available() → bool

Returns whether this provider is available to use in this environment.

provided_transforms() → Iterable[str]

Returns a list of transform type names this provider can handle.

create_transform(typ: str, args: Mapping[str, Any], yaml_create_transform: Callable[[Mapping[str, Any], Iterable[apache_beam.pvalue.PCollection]], apache_beam.transforms.ptransform.PTransform]) → apache_beam.transforms.ptransform.PTransform

Creates a PTransform instance for the given transform type and arguments.

affinity(other: apache_beam.yaml.yaml_provider.Provider)

Returns a value approximating how good it would be for this provider to be used immediately following a transform from the other provider (e.g. to encourage fusion).

apache_beam.yaml.yaml_provider.as_provider(name, provider_or_constructor)

apache_beam.yaml.yaml_provider.as_provider_list(name, lst)

class apache_beam.yaml.yaml_provider.ExternalProvider(urns, service)

Bases: apache_beam.yaml.yaml_provider.Provider

A Provider implemented via the cross language transform service.

provided_transforms()

create_transform(type, args, yaml_create_transform)

create_external_transform(urn, args)

classmethod provider_from_spec(spec)

classmethod register_provider_type(type_name)

class apache_beam.yaml.yaml_provider.ExternalJavaProvider(urns, jar_provider)

Bases: apache_beam.yaml.yaml_provider.ExternalProvider

available()

apache_beam.yaml.yaml_provider.fix_pycallable()

class apache_beam.yaml.yaml_provider.InlineProvider(transform_factories)

Bases: apache_beam.yaml.yaml_provider.Provider

available()

provided_transforms()

create_transform(type, args, yaml_create_transform)

to_json()

class apache_beam.yaml.yaml_provider.MetaInlineProvider(transform_factories)

Bases: apache_beam.yaml.yaml_provider.InlineProvider

create_transform(type, args, yaml_create_transform)

apache_beam.yaml.yaml_provider.create_builtin_provider()

class apache_beam.yaml.yaml_provider.PypiExpansionService(packages, base_python='/home/runner/work/beam/beam/beam/sdks/python/target/.tox/py38-docs/bin/python')

Bases: object

Expands transforms by fully qualified name in a virtual environment with the given dependencies.

VENV_CACHE = '/home/runner/.apache_beam/cache/venvs'

apache_beam.yaml.yaml_provider.parse_providers(provider_specs)

apache_beam.yaml.yaml_provider.merge_providers(*provider_sets)

apache_beam.yaml.yaml_provider.standard_providers()