Package org.apache.beam.sdk.io.sparkreceiver

Class SparkReceiverIO.Read<V>

java.lang.Object
org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<V>>
org.apache.beam.sdk.io.sparkreceiver.SparkReceiverIO.Read<V>
All Implemented Interfaces:
Serializable, HasDisplayData
Enclosing class:
SparkReceiverIO
public abstract static class SparkReceiverIO.Read<V> extends PTransform<PBegin,PCollection<V>>
A PTransform to read from Spark Receiver.
See Also:

  • Constructor Details

    • Read

      public Read()

  • Method Details

    • withSparkReceiverBuilder

      public SparkReceiverIO.Read<V> withSparkReceiverBuilder(ReceiverBuilder<V,? extends org.apache.spark.streaming.receiver.Receiver<V>> sparkReceiverBuilder)
      Sets ReceiverBuilder with value and custom Spark Receiver class.

    • withGetOffsetFn

      public SparkReceiverIO.Read<V> withGetOffsetFn(SerializableFunction<V,Long> getOffsetFn)
      A function to get offset in order to start Receiver from it.

    • withTimestampFn

      public SparkReceiverIO.Read<V> withTimestampFn(SerializableFunction<V,Instant> timestampFn)
      A function to calculate timestamp for a record.

    • withPullFrequencySec

      public SparkReceiverIO.Read<V> withPullFrequencySec(Long pullFrequencySec)
      Delay in seconds between polling for new records updates.

    • withStartPollTimeoutSec

      public SparkReceiverIO.Read<V> withStartPollTimeoutSec(Long startPollTimeoutSec)
      Waiting time after the Receiver starts. Required to prepare for polling.

    • withStartOffset

      public SparkReceiverIO.Read<V> withStartOffset(Long startOffset)
      Inclusive start offset from which the reading should be started.

    • expand

      public PCollection<V> expand(PBegin input)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PBegin,PCollection<V>>

    • validateTransform

      public void validateTransform()