org.apache.beam.sdk.io.gcp.pubsublite

Class PubsubLiteIO

java.lang.Object

org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO

public final class PubsubLiteIO
extends java.lang.Object

I/O transforms for reading from Google Pub/Sub Lite.

For the differences between this and Google Pub/Sub, please refer to the product documentation.

Updates to the I/O connector code

For any significant updates to this I/O connector, please consider involving corresponding code reviewers mentioned here.

Method Summary

Modifier and Type	Method and Description
static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>>	addUuids() Add Uuids to to-be-published messages that ensures that uniqueness is maintained.
static PTransform<PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>>	deduplicate(UuidDeduplicationOptions options) Remove duplicates from the PTransform from a read.
static PTransform<PBegin,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>>	read(SubscriberOptions options) Read messages from Pub/Sub Lite.
static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PDone>	write(PublisherOptions options) Write messages to Pub/Sub Lite.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

read

public static PTransform<PBegin,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>> read(SubscriberOptions options)

Read messages from Pub/Sub Lite. These messages may contain duplicates if the publisher retried, which the PubsubLiteIO write method will do. Use the dedupe transform to remove these duplicates.

Note that this will not pick up newly added partitions. To pick up new partitions, drain and restart the pipeline.

 Pipeline p = ...;

 SubscriptionPath subscriptionPath =
         SubscriptionPath.newBuilder()
             .setLocation(zone)
             .setProjectNumber(projectNum)
             .setName(subscriptionName)
             .build();

 PCollection<SequencedMessage> messages = p.apply(PubsubLiteIO.read(SubscriberOptions.newBuilder()
     .setSubscriptionPath(subscriptionPath)
     .build()), "read");
 

deduplicate

public static PTransform<PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>> deduplicate(UuidDeduplicationOptions options)

Remove duplicates from the PTransform from a read. Assumes by default that the uuids were added by a call to PubsubLiteIO.addUuids() when published.

 PCollection<SequencedMessage> messages = ... (above) ...;
 messages = messages.apply(PubsubLiteIO.deduplicate(
     UuidDeduplicationOptions.newBuilder().build()));

 

addUuids

public static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>> addUuids()

Add Uuids to to-be-published messages that ensures that uniqueness is maintained.

 PCollection<Message> messages = ...;
 messages = messages.apply(PubsubLiteIO.addUuids());

 

write

public static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PDone> write(PublisherOptions options)

Write messages to Pub/Sub Lite.

 TopicPath topicPath =
         TopicPath.newBuilder()
             .setProjectNumber(projectNum)
             .setLocation(zone)
             .setName(topicName)
             .build();

 PCollection<Message> messages = ...;
 messages.apply(PubsubLiteIO.write(
     PublisherOptions.newBuilder().setTopicPath(topicPath).build());