org.apache.beam.runners.spark.io

Class SparkUnboundedSource

java.lang.Object

org.apache.beam.runners.spark.io.SparkUnboundedSource

public class SparkUnboundedSource
extends java.lang.Object

A "composite" InputDStream implementation for UnboundedSources.

This read is a composite of the following steps:

• Create a single-element (per-partition) stream, that contains the (partitioned) Source and an optional UnboundedSource.CheckpointMark to start from.
• Read from within a stateful operation JavaPairDStream.mapWithState(StateSpec) using the StateSpecFunctions.mapSourceFunction(org.apache.beam.runners.spark.translation.SparkRuntimeContext, java.lang.String) mapping function, which manages the state of the CheckpointMark per partition.
• Since the stateful operation is a map operation, the read iterator needs to be flattened, while reporting the properties of the read (such as number of records) to the tracker.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SparkUnboundedSource.Metadata A metadata holder for an input stream partition.

Constructor Summary

Constructors

Constructor and Description
SparkUnboundedSource()

Method Summary

Modifier and Type	Method and Description
static <T,CheckpointMarkT extends UnboundedSource.CheckpointMark> org.apache.beam.runners.spark.translation.streaming.UnboundedDataset<T>	read(org.apache.spark.streaming.api.java.JavaStreamingContext jssc, org.apache.beam.runners.spark.translation.SparkRuntimeContext rc, UnboundedSource<T,CheckpointMarkT> source, java.lang.String stepName)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkUnboundedSource

public SparkUnboundedSource()

Method Detail

read

public static <T,CheckpointMarkT extends UnboundedSource.CheckpointMark> org.apache.beam.runners.spark.translation.streaming.UnboundedDataset<T> read(org.apache.spark.streaming.api.java.JavaStreamingContext jssc,
                                                                                                                                                      org.apache.beam.runners.spark.translation.SparkRuntimeContext rc,
                                                                                                                                                      UnboundedSource<T,CheckpointMarkT> source,
                                                                                                                                                      java.lang.String stepName)