public class SparkReceiverIO
extends java.lang.Object
Receiver.
SparkReceiverIOYou will need to pass a ReceiverBuilder which is responsible for instantiating new
Receiver objects.
Receiver that will be used should implement HasOffset interface. You will need
to pass getOffsetFn which is a SerializableFunction that defines how to get
Long offset from V record.
Optionally you can pass timestampFn which is a SerializableFunction that
defines how to get Instant timestamp from V record, you can pass startOffset which is inclusive start offset from which the reading should be started.
Optionally you can pass pullFrequencySec which is a delay in seconds between polling
for new records updates. Also, you can pass startPollTimeoutSec which is delay in seconds
before start polling.
Example of read() usage:
Pipeline p = ...; // Create pipeline.
// Create ReceiverBuilder for CustomReceiver
ReceiverBuilder<String, CustomReceiverWithOffset> receiverBuilder =
new ReceiverBuilder<>(CustomReceiver.class).withConstructorArgs();
//Read from CustomReceiver
p.apply("Spark Receiver Read",
SparkReceiverIO.Read<String> reader =
SparkReceiverIO.<String>read()
.withGetOffsetFn(Long::valueOf)
.withTimestampFn(Instant::parse)
.withPullFrequencySec(1L)
.withStartPollTimeoutSec(2L)
.withStartOffset(10L)
.withSparkReceiverBuilder(receiverBuilder);
| Modifier and Type | Class and Description |
|---|---|
static class |
SparkReceiverIO.Read<V>
A
PTransform to read from Spark Receiver. |
| Constructor and Description |
|---|
SparkReceiverIO() |
| Modifier and Type | Method and Description |
|---|---|
static <V> SparkReceiverIO.Read<V> |
read() |
public static <V> SparkReceiverIO.Read<V> read()