org.apache.beam.sdk.io.sparkreceiver

Class SparkReceiverIO.Read<V>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<V>>

org.apache.beam.sdk.io.sparkreceiver.SparkReceiverIO.Read<V>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

SparkReceiverIO

public abstract static class SparkReceiverIO.Read<V>
extends PTransform<PBegin,PCollection<V>>

A PTransform to read from Spark Receiver.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Read()

Method Summary

Modifier and Type	Method and Description
PCollection<V>	expand(PBegin input) Override this method to specify how this PTransform should be expanded on the given InputT.
void	validateTransform()
SparkReceiverIO.Read<V>	withGetOffsetFn(SerializableFunction<V,java.lang.Long> getOffsetFn) A function to get offset in order to start Receiver from it.
SparkReceiverIO.Read<V>	withPullFrequencySec(java.lang.Long pullFrequencySec) Delay in seconds between polling for new records updates.
SparkReceiverIO.Read<V>	withSparkReceiverBuilder(ReceiverBuilder<V,? extends org.apache.spark.streaming.receiver.Receiver<V>> sparkReceiverBuilder) Sets ReceiverBuilder with value and custom Spark Receiver class.
SparkReceiverIO.Read<V>	withStartOffset(java.lang.Long startOffset) Inclusive start offset from which the reading should be started.
SparkReceiverIO.Read<V>	withStartPollTimeoutSec(java.lang.Long startPollTimeoutSec) Waiting time after the Receiver starts.
SparkReceiverIO.Read<V>	withTimestampFn(SerializableFunction<V,Instant> timestampFn) A function to calculate timestamp for a record.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Read

public Read()

Method Detail

withSparkReceiverBuilder

public SparkReceiverIO.Read<V> withSparkReceiverBuilder(ReceiverBuilder<V,? extends org.apache.spark.streaming.receiver.Receiver<V>> sparkReceiverBuilder)

Sets ReceiverBuilder with value and custom Spark Receiver class.

withGetOffsetFn

public SparkReceiverIO.Read<V> withGetOffsetFn(SerializableFunction<V,java.lang.Long> getOffsetFn)

A function to get offset in order to start Receiver from it.

withTimestampFn

public SparkReceiverIO.Read<V> withTimestampFn(SerializableFunction<V,Instant> timestampFn)

A function to calculate timestamp for a record.

withPullFrequencySec

public SparkReceiverIO.Read<V> withPullFrequencySec(java.lang.Long pullFrequencySec)

Delay in seconds between polling for new records updates.

withStartPollTimeoutSec

public SparkReceiverIO.Read<V> withStartPollTimeoutSec(java.lang.Long startPollTimeoutSec)

Waiting time after the Receiver starts. Required to prepare for polling.

withStartOffset

public SparkReceiverIO.Read<V> withStartOffset(java.lang.Long startOffset)

Inclusive start offset from which the reading should be started.

expand

public PCollection<V> expand(PBegin input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PBegin,PCollection<V>>

validateTransform

public void validateTransform()