Package org.apache.beam.runners.spark.translation

Class SingleEmitInputDStream<T>

java.lang.Object
org.apache.spark.streaming.dstream.DStream<T>
org.apache.spark.streaming.dstream.InputDStream<T>
org.apache.spark.streaming.dstream.ConstantInputDStream<T>
org.apache.beam.runners.spark.translation.SingleEmitInputDStream<T>
Type Parameters:
T - The type of elements in the RDD
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, scala.Serializable
public class SingleEmitInputDStream<T> extends org.apache.spark.streaming.dstream.ConstantInputDStream<T>
A specialized ConstantInputDStream that emits its RDD exactly once. Alternative to QueueInputDStream when checkpointing is required.

Features:

  • Supports checkpointing
  • Guarantees single emission of data
  • Returns empty RDD after first emission
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    SingleEmitInputDStream(org.apache.spark.streaming.StreamingContext ssc, org.apache.spark.rdd.RDD<T> rdd)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    scala.Option<org.apache.spark.rdd.RDD<T>>
    compute(org.apache.spark.streaming.Time validTime)
     

    Methods inherited from class org.apache.spark.streaming.dstream.ConstantInputDStream

    start, stop

    Methods inherited from class org.apache.spark.streaming.dstream.InputDStream

    baseScope, dependencies, id, isTimeValid, lastValidTime, lastValidTime_$eq, name, rateController, slideDuration

    Methods inherited from class org.apache.spark.streaming.dstream.DStream

    cache, checkpoint, checkpointData, checkpointDuration, checkpointDuration_$eq, clearCheckpointData, clearMetadata, context, count, countByValue, countByValue$default$1, countByValue$default$2, countByValueAndWindow, countByValueAndWindow$default$3, countByValueAndWindow$default$4, countByWindow, createRDDWithLocalProperties, creationSite, filter, flatMap, foreachRDD, foreachRDD, generatedRDDs, generatedRDDs_$eq, generateJob, getOrCompute, glom, graph, graph_$eq, initialize, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isInitialized, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, map, mapPartitions, mapPartitions$default$2, mustCheckpoint, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, parentRememberDuration, persist, persist, print, print, reduce, reduceByWindow, reduceByWindow, register, remember, rememberDuration, rememberDuration_$eq, repartition, restoreCheckpointData, saveAsObjectFiles, saveAsObjectFiles$default$2, saveAsTextFiles, saveAsTextFiles$default$2, setContext, setGraph, slice, slice, ssc, ssc_$eq, storageLevel, storageLevel_$eq, toPairDStreamFunctions, toPairDStreamFunctions$default$4, transform, transform, transformWith, transformWith, union, updateCheckpointData, validateAtStart, window, window, zeroTime, zeroTime_$eq

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • SingleEmitInputDStream

      public SingleEmitInputDStream(org.apache.spark.streaming.StreamingContext ssc, org.apache.spark.rdd.RDD<T> rdd)

  • Method Details

    • compute

      public scala.Option<org.apache.spark.rdd.RDD<T>> compute(org.apache.spark.streaming.Time validTime)
      Overrides:
      compute in class org.apache.spark.streaming.dstream.ConstantInputDStream<T>