org.apache.beam.runners.spark.stateful

Class StateSpecFunctions

java.lang.Object

org.apache.beam.runners.spark.stateful.StateSpecFunctions

public class StateSpecFunctions
extends java.lang.Object

A class containing StateSpec mappingFunctions.

Constructor Summary

Constructors

Constructor and Description
StateSpecFunctions()

Method Summary

Modifier and Type	Method and Description
static <T,CheckpointMarkT extends UnboundedSource.CheckpointMark> scala.Function3<Source<T>,scala.Option<CheckpointMarkT>,org.apache.spark.streaming.State<scala.Tuple2<byte[],Instant>>,scala.Tuple2<java.lang.Iterable<byte[]>,SparkUnboundedSource.Metadata>>	mapSourceFunction(org.apache.beam.runners.core.construction.SerializablePipelineOptions options, java.lang.String stepName) A StateSpec function to support reading from an UnboundedSource.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StateSpecFunctions

public StateSpecFunctions()

Method Detail

mapSourceFunction

public static <T,CheckpointMarkT extends UnboundedSource.CheckpointMark> scala.Function3<Source<T>,scala.Option<CheckpointMarkT>,org.apache.spark.streaming.State<scala.Tuple2<byte[],Instant>>,scala.Tuple2<java.lang.Iterable<byte[]>,SparkUnboundedSource.Metadata>> mapSourceFunction(org.apache.beam.runners.core.construction.SerializablePipelineOptions options,
                                                                                                                                                                                                                                                                                          java.lang.String stepName)

A StateSpec function to support reading from an UnboundedSource.

This StateSpec function expects the following:

• Key: The (partitioned) Source to read from.
• Value: An optional UnboundedSource.CheckpointMark to start from.
• State: A byte representation of the (previously) persisted CheckpointMark.

And returns an iterator over all read values (for the micro-batch).

This stateful operation could be described as a flatMap over a single-element stream, which outputs all the elements read from the UnboundedSource for this micro-batch. Since micro-batches are bounded, the provided UnboundedSource is wrapped by a MicrobatchSource that applies bounds in the form of duration and max records (per micro-batch).

In order to avoid using Spark Guava's classes which pollute the classpath, we use the StateSpec.function(scala.Function3) signature which employs scala's native Option, instead of the StateSpec.function(org.apache.spark.api.java.function.Function3) signature, which employs Guava's Optional.

See also SPARK-4819.

Type Parameters:

T - The type of the input stream elements.

CheckpointMarkT - The type of the UnboundedSource.CheckpointMark.

Parameters:

options - A serializable SerializablePipelineOptions.

Returns:

The appropriate StateSpec function.