apache_beam.runners.sdf_common module

This module contains Splittable DoFn logic that’s common to all runners.

class apache_beam.runners.sdf_common.SplittableParDoOverride

Bases: apache_beam.pipeline.PTransformOverride

A transform override for ParDo transformss of SplittableDoFns.

Replaces the ParDo transform with a SplittableParDo transform that performs SDF specific logic.

matches(applied_ptransform)

get_replacement_transform(ptransform)

class apache_beam.runners.sdf_common.SplittableParDo(ptransform)

Bases: apache_beam.transforms.ptransform.PTransform

A transform that processes a PCollection using a Splittable DoFn.

expand(pcoll)

class apache_beam.runners.sdf_common.ElementAndRestriction(element, restriction)

Bases: future.types.newobject.newobject

A holder for an element and a restriction.

class apache_beam.runners.sdf_common.PairWithRestrictionFn(do_fn)

Bases: apache_beam.transforms.core.DoFn

A transform that pairs each element with a restriction.

start_bundle()

process(element, window=WindowParam, *args, **kwargs)

class apache_beam.runners.sdf_common.SplitRestrictionFn(do_fn)

Bases: apache_beam.transforms.core.DoFn

A transform that perform initial splitting of Splittable DoFn inputs.

start_bundle()

process(element_and_restriction, *args, **kwargs)

class apache_beam.runners.sdf_common.ExplodeWindowsFn(*unused_args, **unused_kwargs)

Bases: apache_beam.transforms.core.DoFn

A transform that forces the runner to explode windows.

This is done to make sure that Splittable DoFn proceses an element for each of the windows that element belongs to.

process(element, window=WindowParam, *args, **kwargs)

class apache_beam.runners.sdf_common.RandomUniqueKeyFn(*unused_args, **unused_kwargs)

Bases: apache_beam.transforms.core.DoFn

A transform that assigns a unique key to each element.

process(element, window=WindowParam, *args, **kwargs)

class apache_beam.runners.sdf_common.ProcessKeyedElements(sdf, element_coder, restriction_coder, windowing_strategy, ptransform_args, ptransform_kwargs, ptransform_side_inputs)

Bases: apache_beam.transforms.ptransform.PTransform

A primitive transform that performs SplittableDoFn magic.

Input to this transform should be a PCollection of keyed ElementAndRestriction objects.

expand(pcoll)