apache_beam.yaml.yaml_provider module

This module defines Providers usable from yaml, which is a specification for where to find and how to invoke services that vend implementations of various PTransforms.

class apache_beam.yaml.yaml_provider.Provider

Bases: object

Maps transform types names and args to concrete PTransform instances.

available() → bool

Returns whether this provider is available to use in this environment.

cache_artifacts() → Optional[Iterable[str]]

provided_transforms() → Iterable[str]

Returns a list of transform type names this provider can handle.

config_schema(type)

description(type)

requires_inputs(typ: str, args: Mapping[str, Any]) → bool

Returns whether this transform requires inputs.

Specifically, if this returns True and inputs are not provided than an error will be thrown.

This is best-effort, primarily for better and earlier error messages.

create_transform(typ: str, args: Mapping[str, Any], yaml_create_transform: Callable[[Mapping[str, Any], Iterable[apache_beam.pvalue.PCollection]], apache_beam.transforms.ptransform.PTransform]) → apache_beam.transforms.ptransform.PTransform

Creates a PTransform instance for the given transform type and arguments.

underlying_provider()

If this provider is simply a proxy to another provider, return the provider that should actually be used for affinity checking.

affinity(other: apache_beam.yaml.yaml_provider.Provider)

Returns a value approximating how good it would be for this provider to be used immediately following a transform from the other provider (e.g. to encourage fusion).

apache_beam.yaml.yaml_provider.as_provider(name, provider_or_constructor)

apache_beam.yaml.yaml_provider.as_provider_list(name, lst)

class apache_beam.yaml.yaml_provider.ExternalProvider(urns, service)

Bases: apache_beam.yaml.yaml_provider.Provider

A Provider implemented via the cross language transform service.

provided_transforms()

schema_transforms()

config_schema(type)

description(type)

requires_inputs(typ, args)

create_transform(type, args, yaml_create_transform)

create_external_transform(urn, args)

classmethod provider_from_spec(spec)

classmethod register_provider_type(type_name)

apache_beam.yaml.yaml_provider.java_jar(urns, jar: str)

apache_beam.yaml.yaml_provider.maven_jar(urns, *, artifact_id, group_id, version, repository='https://repo.maven.apache.org/maven2', classifier=None, appendix=None)

apache_beam.yaml.yaml_provider.beam_jar(urns, *, gradle_target, appendix=None, version='2.53.0', artifact_id=None)

apache_beam.yaml.yaml_provider.docker(urns, **config)

class apache_beam.yaml.yaml_provider.RemoteProvider(urns, address: str)

Bases: apache_beam.yaml.yaml_provider.ExternalProvider

available()

cache_artifacts()

class apache_beam.yaml.yaml_provider.ExternalJavaProvider(urns, jar_provider)

Bases: apache_beam.yaml.yaml_provider.ExternalProvider

available()

cache_artifacts()

apache_beam.yaml.yaml_provider.python(urns, packages=())

class apache_beam.yaml.yaml_provider.ExternalPythonProvider(urns, packages)

Bases: apache_beam.yaml.yaml_provider.ExternalProvider

available()

cache_artifacts()

create_external_transform(urn, args)

apache_beam.yaml.yaml_provider.fix_pycallable()

class apache_beam.yaml.yaml_provider.InlineProvider(transform_factories, no_input_transforms=())

Bases: apache_beam.yaml.yaml_provider.Provider

available()

cache_artifacts()

provided_transforms()

config_schema(typ)

description(typ)

get_docs(typ)

create_transform(type, args, yaml_create_transform)

to_json()

requires_inputs(typ, args)

class apache_beam.yaml.yaml_provider.MetaInlineProvider(transform_factories, no_input_transforms=())

Bases: apache_beam.yaml.yaml_provider.InlineProvider

create_transform(type, args, yaml_create_transform)

class apache_beam.yaml.yaml_provider.SqlBackedProvider(transforms: Mapping[str, Callable[[...], apache_beam.transforms.ptransform.PTransform]], sql_provider: Optional[apache_beam.yaml.yaml_provider.Provider] = None)

Bases: apache_beam.yaml.yaml_provider.Provider

sql_provider()

provided_transforms()

available()

cache_artifacts()

underlying_provider()

to_json()

create_transform(typ: str, args: Mapping[str, Any], yaml_create_transform: Any) → apache_beam.transforms.ptransform.PTransform

apache_beam.yaml.yaml_provider.element_to_rows(e)

apache_beam.yaml.yaml_provider.dicts_to_rows(o)

apache_beam.yaml.yaml_provider.create_builtin_provider()

class apache_beam.yaml.yaml_provider.PypiExpansionService(packages, base_python='/home/runner/work/beam/beam/beam/sdks/python/target/.tox/py38-docs/bin/python')

Bases: object

Expands transforms by fully qualified name in a virtual environment with the given dependencies.

VENV_CACHE = '/home/runner/.apache_beam/cache/venvs'

class apache_beam.yaml.yaml_provider.RenamingProvider(transforms, mappings, underlying_provider, defaults=None)

Bases: apache_beam.yaml.yaml_provider.Provider

static expand_mappings(mappings)

available() → bool

provided_transforms() → Iterable[str]

config_schema(type)

description(typ)

requires_inputs(typ, args)

create_transform(typ: str, args: Mapping[str, Any], yaml_create_transform: Callable[[Mapping[str, Any], Iterable[apache_beam.pvalue.PCollection]], apache_beam.transforms.ptransform.PTransform]) → apache_beam.transforms.ptransform.PTransform

Creates a PTransform instance for the given transform type and arguments.

underlying_provider()

cache_artifacts()

apache_beam.yaml.yaml_provider.parse_providers(provider_specs)

apache_beam.yaml.yaml_provider.merge_providers(*provider_sets)

apache_beam.yaml.yaml_provider.standard_providers()