Package org.apache.beam.runners.spark.translation

Class AbstractInOutIterator<K,InputT,OutputT>

java.lang.Object
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.UnmodifiableIterator<scala.Tuple2<TupleTag<?>,WindowedValue<?>>>
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.AbstractIterator<scala.Tuple2<TupleTag<?>,WindowedValue<?>>>
org.apache.beam.runners.spark.translation.AbstractInOutIterator<K,InputT,OutputT>
Type Parameters:
K - The key type for the processing context
InputT - The input element type to be processed
OutputT - The output element type after processing
All Implemented Interfaces:
Iterator<scala.Tuple2<TupleTag<?>,WindowedValue<?>>>
public abstract class AbstractInOutIterator<K,InputT,OutputT> extends org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.AbstractIterator<scala.Tuple2<TupleTag<?>,WindowedValue<?>>>
Abstract base class for iterators that process Spark input data and produce corresponding output values. This class serves as a common base for both bounded and unbounded processing strategies in the Spark runner.

The class extends Guava's AbstractIterator and provides common functionality for iterating through input elements, processing them using a DoFnRunner, and producing output elements as tuples of TupleTag and WindowedValue pairs.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected final SparkProcessContext<K,InputT,OutputT>
    ctx
     

  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    AbstractInOutIterator(SparkProcessContext<K,InputT,OutputT> ctx)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    fireTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)
    Fires a timer using the DoFnRunner from the context and performs cleanup afterwards.

    Methods inherited from class org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.AbstractIterator

    computeNext, endOfData, hasNext, next, peek

    Methods inherited from class org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.UnmodifiableIterator

    remove

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface java.util.Iterator

    forEachRemaining

  • Field Details

  • Constructor Details

  • Method Details

    • fireTimer

      public void fireTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)
      Fires a timer using the DoFnRunner from the context and performs cleanup afterwards.

      After firing the timer, if the timer data iterator is an instance of ParDoStateUpdateFn.SparkTimerInternalsIterator, the fired timer will be deleted as part of cleanup to prevent re-firing of the same timer.

      Parameters:
      timer - The timer data containing information about the timer to fire
      Throws:
      IllegalArgumentException - If the timer namespace is not a StateNamespaces.WindowNamespace