Package org.apache.beam.sdk.io.gcp.pubsub

Class PubsubUnboundedSink

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<PubsubMessage>,PDone>

org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink

All Implemented Interfaces:

Serializable, HasDisplayData

public class PubsubUnboundedSink
extends PTransform<PCollection<PubsubMessage>,PDone>

A PTransform which streams messages to Pubsub.

• The underlying implementation is just a GroupByKey followed by a ParDo which publishes as a side effect. (In the future we want to design and switch to a custom UnboundedSink implementation so as to gain access to system watermark and end-of-pipeline cleanup.)
• We try to send messages in batches while also limiting send latency.
• No stats are logged. Rather some counters are used to keep track of elements and batches.
• Though some background threads are used by the underlying netty system all actual Pubsub calls are blocking. We rely on the underlying runner to allow multiple DoFn instances to execute concurrently and hide latency.
• A failed bundle will cause messages to be resent. Thus we rely on the Pubsub consumer to dedup messages.

See Also:

• Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor	Description
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, String timestampAttribute, String idAttribute, int numShards, boolean publishBatchWithOrderingKey)
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, String timestampAttribute, String idAttribute, int numShards, boolean publishBatchWithOrderingKey, int publishBatchSize, int publishBatchBytes)
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, String timestampAttribute, String idAttribute, int numShards, boolean publishBatchWithOrderingKey, int publishBatchSize, int publishBatchBytes, String pubsubRootUrl)
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, String timestampAttribute, String idAttribute, int numShards, boolean publishBatchWithOrderingKey, String pubsubRootUrl)

Method Summary

Modifier and Type	Method	Description
PDone	expand(PCollection<PubsubMessage> input)	Override this method to specify how this PTransform should be expanded on the given InputT.
@Nullable String	getIdAttribute()	Get the id attribute.
boolean	getPublishBatchWithOrderingKey()
@Nullable String	getTimestampAttribute()	Get the timestamp attribute.
@Nullable PubsubClient.TopicPath	getTopic()	Get the topic being written to.
@Nullable ValueProvider<PubsubClient.TopicPath>	getTopicProvider()	Get the ValueProvider for the topic being written to.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
 ValueProvider<PubsubClient.TopicPath> topic,
 String timestampAttribute,
 String idAttribute,
 int numShards,
 boolean publishBatchWithOrderingKey)

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
 ValueProvider<PubsubClient.TopicPath> topic,
 String timestampAttribute,
 String idAttribute,
 int numShards,
 boolean publishBatchWithOrderingKey,
 String pubsubRootUrl)

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
 ValueProvider<PubsubClient.TopicPath> topic,
 String timestampAttribute,
 String idAttribute,
 int numShards,
 boolean publishBatchWithOrderingKey,
 int publishBatchSize,
 int publishBatchBytes)

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
 ValueProvider<PubsubClient.TopicPath> topic,
 String timestampAttribute,
 String idAttribute,
 int numShards,
 boolean publishBatchWithOrderingKey,
 int publishBatchSize,
 int publishBatchBytes,
 String pubsubRootUrl)

Method Details

getTopic

public @Nullable PubsubClient.TopicPath getTopic()

Get the topic being written to.

getTopicProvider

public @Nullable ValueProvider<PubsubClient.TopicPath> getTopicProvider()

Get the ValueProvider for the topic being written to.

getTimestampAttribute

public @Nullable String getTimestampAttribute()

Get the timestamp attribute.

getIdAttribute

public @Nullable String getIdAttribute()

Get the id attribute.

getPublishBatchWithOrderingKey

public boolean getPublishBatchWithOrderingKey()

expand

public PDone expand(PCollection<PubsubMessage> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<PubsubMessage>,PDone>