Package org.apache.beam.sdk.io.sparkreceiver

Class SparkReceiverIO

java.lang.Object
org.apache.beam.sdk.io.sparkreceiver.SparkReceiverIO
public class SparkReceiverIO extends Object
Streaming sources for Spark Receiver.

Reading using SparkReceiverIO

You will need to pass a ReceiverBuilder which is responsible for instantiating new Receiver objects.

Receiver that will be used should implement HasOffset interface. You will need to pass getOffsetFn which is a SerializableFunction that defines how to get Long offset from V record.

Optionally you can pass timestampFn which is a SerializableFunction that defines how to get Instant timestamp from V record, you can pass startOffset which is inclusive start offset from which the reading should be started.

Optionally you can pass pullFrequencySec which is a delay in seconds between polling for new records updates. Also, you can pass startPollTimeoutSec which is delay in seconds before start polling.

Example of read() usage: 


 Pipeline p = ...; // Create pipeline.

 // Create ReceiverBuilder for CustomReceiver
 ReceiverBuilder<String, CustomReceiverWithOffset> receiverBuilder =
         new ReceiverBuilder<>(CustomReceiver.class).withConstructorArgs();

 //Read from CustomReceiver
 p.apply("Spark Receiver Read",
  SparkReceiverIO.Read<String> reader =
    SparkReceiverIO.<String>read()
      .withGetOffsetFn(Long::valueOf)
      .withTimestampFn(Instant::parse)
      .withPullFrequencySec(1L)
      .withStartPollTimeoutSec(2L)
      .withStartOffset(10L)
      .withSparkReceiverBuilder(receiverBuilder);

  • Constructor Details

    • SparkReceiverIO

      public SparkReceiverIO()

  • Method Details