Package org.apache.beam.runners.spark.translation.streaming

Class ParDoStateUpdateFn<KeyT,ValueT,InputT extends KV<KeyT,ValueT>,OutputT>

java.lang.Object

scala.runtime.AbstractFunction3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,List<scala.Tuple2<TupleTag<?>,byte[]>>>

org.apache.beam.runners.spark.translation.streaming.ParDoStateUpdateFn<KeyT,ValueT,InputT,OutputT>

Type Parameters:

KeyT - The type of the key in the input KV pairs

ValueT - The type of the value in the input KV pairs

InputT - The input type, must be a KV of KeyT and ValueT

OutputT - The output type produced by the DoFn

All Implemented Interfaces:

Serializable, scala.Function3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,List<scala.Tuple2<TupleTag<?>,byte[]>>>

public class ParDoStateUpdateFn<KeyT,ValueT,InputT extends KV<KeyT,ValueT>,OutputT>
extends scala.runtime.AbstractFunction3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,List<scala.Tuple2<TupleTag<?>,byte[]>>>
implements Serializable

A function to handle stateful processing in Apache Beam's SparkRunner. This class processes stateful DoFn operations by managing state updates in a Spark streaming context.

Current Implementation Status:

• State: Fully implemented and supported through SparkStateInternals
• Timers: Processing time timers are now supported through SparkTimerInternals.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ParDoStateUpdateFn.SparkTimerInternalsIterator	An iterator implementation that processes timers from SparkTimerInternals.

Constructor Summary

Constructors

Constructor

Description

ParDoStateUpdateFn(MetricsContainerStepMapAccumulator metricsAccum, String stepName, DoFn<InputT,OutputT> doFn, Coder<KeyT> keyCoder, WindowedValues.FullWindowedValueCoder<ValueT> wvCoder, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, TupleTag<?> mainOutputTag, List<TupleTag<?>> additionalOutputTags, Coder<InputT> inputCoder, Map<TupleTag<?>,Coder<?>> outputCoders, Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, WindowingStrategy<?,?> windowingStrategy, DoFnSchemaInformation doFnSchemaInformation, Map<String,PCollectionView<?>> sideInputMapping, Map<Integer,GlobalWatermarkHolder.SparkWatermarks> watermarks, List<Integer> sourceIds, boolean useStreamingSideInput)

Method Summary

Modifier and Type	Method	Description
List<scala.Tuple2<TupleTag<?>,byte[]>>	apply(ByteArray serializedKey, scala.Option<byte[]> serializedValue, org.apache.spark.streaming.State<StateAndTimers> state)

Methods inherited from class scala.runtime.AbstractFunction3

curried, toString, tupled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

ParDoStateUpdateFn

public ParDoStateUpdateFn(MetricsContainerStepMapAccumulator metricsAccum,
 String stepName,
 DoFn<InputT,OutputT> doFn,
 Coder<KeyT> keyCoder,
 WindowedValues.FullWindowedValueCoder<ValueT> wvCoder,
 org.apache.beam.runners.core.construction.SerializablePipelineOptions options,
 TupleTag<?> mainOutputTag,
 List<TupleTag<?>> additionalOutputTags,
 Coder<InputT> inputCoder,
 Map<TupleTag<?>,Coder<?>> outputCoders,
 Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs,
 WindowingStrategy<?,?> windowingStrategy,
 DoFnSchemaInformation doFnSchemaInformation,
 Map<String,PCollectionView<?>> sideInputMapping,
 Map<Integer,GlobalWatermarkHolder.SparkWatermarks> watermarks,
 List<Integer> sourceIds,
 boolean useStreamingSideInput)

Method Details

apply

public List<scala.Tuple2<TupleTag<?>,byte[]>> apply(ByteArray serializedKey,
 scala.Option<byte[]> serializedValue,
 org.apache.spark.streaming.State<StateAndTimers> state)

Specified by:

apply in interface scala.Function3<KeyT,ValueT,InputT extends KV<KeyT,ValueT>,OutputT>