org.apache.beam.sdk.io.gcp.pubsublite

Class PubsubLiteIO

java.lang.Object

org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO

@Experimental
public final class PubsubLiteIO
extends java.lang.Object

I/O transforms for reading from Google Pub/Sub Lite.

For the differences between this and Google Pub/Sub, please refer to the product documentation.

Method Summary

Modifier and Type	Method and Description
static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>>	addUuids() Add Uuids to to-be-published messages that ensures that uniqueness is maintained.
static PTransform<PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>>	deduplicate(UuidDeduplicationOptions options) Remove duplicates from the PTransform from a read.
static PTransform<PBegin,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>>	read(SubscriberOptions options) Read messages from Pub/Sub Lite.
static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PDone>	write(PublisherOptions options) Write messages to Pub/Sub Lite.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

read

public static PTransform<PBegin,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>> read(SubscriberOptions options)

Read messages from Pub/Sub Lite. These messages may contain duplicates if the publisher retried, which the PubsubLiteIO write method will do. Use the dedupe transform to remove these duplicates.

 Pipeline p = ...;

 SubscriptionPath subscriptionPath =
         SubscriptionPath.newBuilder()
             .setLocation(zone)
             .setProjectNumber(projectNum)
             .setName(subscriptionName)
             .build();

 PCollection<SequencedMessage> messages = p.apply(PubsubLiteIO.read(SubscriberOptions.newBuilder()
     .setSubscriptionPath(subscriptionPath)
     .build()), "read");
 

deduplicate

public static PTransform<PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>> deduplicate(UuidDeduplicationOptions options)

Remove duplicates from the PTransform from a read. Assumes by default that the uuids were added by a call to PubsubLiteIO.addUuids() when published.

 PCollection<SequencedMessage> messages = ... (above) ...;
 messages = messages.apply(PubsubLiteIO.deduplicate(
     UuidDeduplicationOptions.newBuilder().build()));

 

addUuids

public static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>> addUuids()

Add Uuids to to-be-published messages that ensures that uniqueness is maintained.

 PCollection<Message> messages = ...;
 messages = messages.apply(PubsubLiteIO.addUuids());

 

write

public static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PDone> write(PublisherOptions options)

Write messages to Pub/Sub Lite.

 TopicPath topicPath =
         TopicPath.newBuilder()
             .setProjectNumber(projectNum)
             .setLocation(zone)
             .setName(topicName)
             .build();

 PCollection<Message> messages = ...;
 messages.apply(PubsubLiteIO.write(
     PublisherOptions.newBuilder().setTopicPath(topicPath).build());