Package org.apache.beam.sdk.io.gcp.pubsublite

Class PubsubLiteIO

java.lang.Object
org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO
public final class PubsubLiteIO extends Object
I/O transforms for reading from Google Pub/Sub Lite.

For the differences between this and Google Pub/Sub, please refer to the product documentation.

Updates to the I/O connector code

For any significant updates to this I/O connector, please consider involving corresponding code reviewers mentioned here.

  • Method Details

    • read

      public static PTransform<PBegin,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>> read(SubscriberOptions options)
      Read messages from Pub/Sub Lite. These messages may contain duplicates if the publisher retried, which the PubsubLiteIO write method will do. Use the dedupe transform to remove these duplicates.

      Note that this will not pick up newly added partitions. To pick up new partitions, drain and restart the pipeline. 

      
 Pipeline p = ...;

 SubscriptionPath subscriptionPath =
         SubscriptionPath.newBuilder()
             .setLocation(zone)
             .setProjectNumber(projectNum)
             .setName(subscriptionName)
             .build();

 PCollection<SequencedMessage> messages = p.apply(PubsubLiteIO.read(SubscriberOptions.newBuilder()
     .setSubscriptionPath(subscriptionPath)
     .build()), "read");

    • deduplicate

      public static PTransform<PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>,PCollection<com.google.cloud.pubsublite.proto.SequencedMessage>> deduplicate(UuidDeduplicationOptions options)
      Remove duplicates from the PTransform from a read. Assumes by default that the uuids were added by a call to PubsubLiteIO.addUuids() when published. 
      
 PCollection<SequencedMessage> messages = ... (above) ...;
 messages = messages.apply(PubsubLiteIO.deduplicate(
     UuidDeduplicationOptions.newBuilder().build()));

    • addUuids

      public static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>> addUuids()
      Add Uuids to to-be-published messages that ensures that uniqueness is maintained. 
      
 PCollection<Message> messages = ...;
 messages = messages.apply(PubsubLiteIO.addUuids());

    • write

      public static PTransform<PCollection<com.google.cloud.pubsublite.proto.PubSubMessage>,PDone> write(PublisherOptions options)
      Write messages to Pub/Sub Lite. 
      
 TopicPath topicPath =
         TopicPath.newBuilder()
             .setProjectNumber(projectNum)
             .setLocation(zone)
             .setName(topicName)
             .build();

 PCollection<Message> messages = ...;
 messages.apply(PubsubLiteIO.write(
     PublisherOptions.newBuilder().setTopicPath(topicPath).build());