apache_beam.runners.dataflow.ptransform_overrides module¶

Ptransform overrides for DataflowRunner.

class apache_beam.runners.dataflow.ptransform_overrides.CreatePTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for Create in streaming mode.

matches(applied_ptransform)[source]¶

get_replacement_transform_for_applied_ptransform(applied_ptransform)[source]¶

class apache_beam.runners.dataflow.ptransform_overrides.ReadPTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for Read(BoundedSource)

matches(applied_ptransform)[source]¶

get_replacement_transform_for_applied_ptransform(applied_ptransform)[source]¶

class apache_beam.runners.dataflow.ptransform_overrides.JrhReadPTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for Read(BoundedSource)

matches(applied_ptransform)[source]¶

get_replacement_transform_for_applied_ptransform(applied_ptransform)[source]¶

class apache_beam.runners.dataflow.ptransform_overrides.CombineValuesPTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for CombineValues.

The DataflowRunner expects that the CombineValues PTransform acts as a primitive. So this override replaces the CombineValues with a primitive.

matches(applied_ptransform)[source]¶

get_replacement_transform(ptransform)[source]¶

class apache_beam.runners.dataflow.ptransform_overrides.NativeReadPTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for Read using native sources.

The DataflowRunner expects that the Read PTransform using native sources act as a primitive. So this override replaces the Read with a primitive.

matches(applied_ptransform)[source]¶

get_replacement_transform(ptransform)[source]¶

class apache_beam.runners.dataflow.ptransform_overrides.WriteToBigQueryPTransformOverride(pipeline, options)[source]¶

Bases: apache_beam.pipeline.PTransformOverride

matches(applied_ptransform)[source]¶

get_replacement_transform(ptransform)[source]¶

class apache_beam.runners.dataflow.ptransform_overrides.GroupIntoBatchesWithShardedKeyPTransformOverride(dataflow_runner, options)[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for GroupIntoBatches.WithShardedKey.

This override simply returns the original transform but additionally records the output PCollection in order to append required step properties during graph translation.

matches(applied_ptransform)[source]¶

get_replacement_transform_for_applied_ptransform(ptransform)[source]¶