Package org.apache.beam.runners.spark.translation.streaming

package org.apache.beam.runners.spark.translation.streaming

Internal utilities to translate Beam pipelines to Spark streaming.

Related Packages

Package	Description
org.apache.beam.runners.spark.translation	Internal translators for running Beam pipelines on Spark.

Classes

Class	Description
Checkpoint	Checkpoint data to make it available in future pipeline runs.
Checkpoint.CheckpointDir	Checkpoint dir tree.
CreateStreamingSparkView<ElemT,ViewT>	Spark streaming overrides for various view (side input) transforms.
CreateStreamingSparkView.CreateSparkPCollectionView<ElemT,ViewT>	Creates a primitive PCollectionView.
CreateStreamingSparkView.Factory<ElemT,ViewT>
ParDoStateUpdateFn<KeyT,ValueT,InputT extends KV<KeyT,ValueT>,OutputT>	A function to handle stateful processing in Apache Beam's SparkRunner.
ParDoStateUpdateFn.SparkTimerInternalsIterator	An iterator implementation that processes timers from SparkTimerInternals.
SparkRunnerStreamingContextFactory	A JavaStreamingContext factory for resilience.
StatefulStreamingParDoEvaluator<KeyT,ValueT,OutputT>	A specialized evaluator for ParDo operations in Spark Streaming context that is invoked when stateful streaming is detected in the DoFn.
StreamingTransformTranslator	Supports translation between a Beam transform, and Spark's operations on DStreams.
StreamingTransformTranslator.SparkTransformsRegistrar	Registers classes specialized by the Spark runner.
StreamingTransformTranslator.Translator	Translator matches Beam transformation with the appropriate evaluator.
TestDStream<T>
UnboundedDataset<T>	DStream holder Can also crate a DStream from a supplied queue of values, but mainly for testing.