apache_beam.runners.dask.transform_evaluator module

Transform Beam PTransforms into Dask Bag operations.

A minimum set of operation substitutions, to adap Beam’s PTransform model to Dask Bag functions.

TODO(alxr): Translate ops from https://docs.dask.org/en/latest/bag-api.html.

class apache_beam.runners.dask.transform_evaluator.DaskBagOp(applied: apache_beam.pipeline.AppliedPTransform)[source]

Bases: abc.ABC

transform
apply(input_bag: Union[dask.bag.core.Bag, Sequence[dask.bag.core.Bag], None]) → dask.bag.core.Bag[source]
class apache_beam.runners.dask.transform_evaluator.NoOp(applied: apache_beam.pipeline.AppliedPTransform)[source]

Bases: apache_beam.runners.dask.transform_evaluator.DaskBagOp

apply(input_bag: Union[dask.bag.core.Bag, Sequence[dask.bag.core.Bag], None]) → dask.bag.core.Bag[source]
class apache_beam.runners.dask.transform_evaluator.Create(applied: apache_beam.pipeline.AppliedPTransform)[source]

Bases: apache_beam.runners.dask.transform_evaluator.DaskBagOp

apply(input_bag: Union[dask.bag.core.Bag, Sequence[dask.bag.core.Bag], None]) → dask.bag.core.Bag[source]
class apache_beam.runners.dask.transform_evaluator.ParDo(applied: apache_beam.pipeline.AppliedPTransform)[source]

Bases: apache_beam.runners.dask.transform_evaluator.DaskBagOp

apply(input_bag: dask.bag.core.Bag) → dask.bag.core.Bag[source]
class apache_beam.runners.dask.transform_evaluator.Map(applied: apache_beam.pipeline.AppliedPTransform)[source]

Bases: apache_beam.runners.dask.transform_evaluator.DaskBagOp

apply(input_bag: dask.bag.core.Bag) → dask.bag.core.Bag[source]
class apache_beam.runners.dask.transform_evaluator.GroupByKey(applied: apache_beam.pipeline.AppliedPTransform)[source]

Bases: apache_beam.runners.dask.transform_evaluator.DaskBagOp

apply(input_bag: dask.bag.core.Bag) → dask.bag.core.Bag[source]
class apache_beam.runners.dask.transform_evaluator.Flatten(applied: apache_beam.pipeline.AppliedPTransform)[source]

Bases: apache_beam.runners.dask.transform_evaluator.DaskBagOp

apply(input_bag: Union[dask.bag.core.Bag, Sequence[dask.bag.core.Bag], None]) → dask.bag.core.Bag[source]