org.apache.beam.runners.dataflow

Class BatchStatefulParDoOverrides

java.lang.Object

org.apache.beam.runners.dataflow.BatchStatefulParDoOverrides

public class BatchStatefulParDoOverrides
extends java.lang.Object

PTransformOverrideFactories that expands to correctly implement stateful ParDo using window-unaware BatchViewOverrides.GroupByKeyAndSortValuesOnly to linearize processing per key.

For the Fn API, the PTransformOverrideFactory is only required to perform per key grouping and expansion.

This implementation relies on implementation details of the Dataflow runner, specifically standard fusion behavior of ParDo tranforms following a GroupByKey.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BatchStatefulParDoOverrides.BatchStatefulDoFn<K,V,OutputT> A key-preserving DoFn that explodes an iterable that has been grouped by key and window.

Constructor Summary

Constructors

Constructor and Description
BatchStatefulParDoOverrides()

Method Summary

Modifier and Type	Method and Description
static <K,InputT,OutputT> org.apache.beam.sdk.runners.PTransformOverrideFactory<PCollection<KV<K,InputT>>,PCollectionTuple,ParDo.MultiOutput<KV<K,InputT>,OutputT>>	multiOutputOverrideFactory(DataflowPipelineOptions options) Returns a PTransformOverrideFactory that replaces a multi-output ParDo with a composite transform specialized for the DataflowRunner.
static <K,InputT,OutputT> org.apache.beam.sdk.runners.PTransformOverrideFactory<PCollection<KV<K,InputT>>,PCollection<OutputT>,ParDo.SingleOutput<KV<K,InputT>,OutputT>>	singleOutputOverrideFactory(DataflowPipelineOptions options) Returns a PTransformOverrideFactory that replaces a single-output ParDo with a composite transform specialized for the DataflowRunner.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BatchStatefulParDoOverrides

public BatchStatefulParDoOverrides()

Method Detail

singleOutputOverrideFactory

public static <K,InputT,OutputT> org.apache.beam.sdk.runners.PTransformOverrideFactory<PCollection<KV<K,InputT>>,PCollection<OutputT>,ParDo.SingleOutput<KV<K,InputT>,OutputT>> singleOutputOverrideFactory(DataflowPipelineOptions options)

Returns a PTransformOverrideFactory that replaces a single-output ParDo with a composite transform specialized for the DataflowRunner.

multiOutputOverrideFactory

public static <K,InputT,OutputT> org.apache.beam.sdk.runners.PTransformOverrideFactory<PCollection<KV<K,InputT>>,PCollectionTuple,ParDo.MultiOutput<KV<K,InputT>,OutputT>> multiOutputOverrideFactory(DataflowPipelineOptions options)

Returns a PTransformOverrideFactory that replaces a multi-output ParDo with a composite transform specialized for the DataflowRunner.