org.apache.beam.sdk.io.gcp.pubsub

Class PubsubUnboundedSink

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<PubsubMessage>,PDone>

org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class PubsubUnboundedSink
extends PTransform<PCollection<PubsubMessage>,PDone>

A PTransform which streams messages to Pubsub.

• The underlying implementation is just a GroupByKey followed by a ParDo which publishes as a side effect. (In the future we want to design and switch to a custom UnboundedSink implementation so as to gain access to system watermark and end-of-pipeline cleanup.)
• We try to send messages in batches while also limiting send latency.
• No stats are logged. Rather some counters are used to keep track of elements and batches.
• Though some background threads are used by the underlying netty system all actual Pubsub calls are blocking. We rely on the underlying runner to allow multiple DoFn instances to execute concurrently and hide latency.
• A failed bundle will cause messages to be resent. Thus we rely on the Pubsub consumer to dedup messages.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, java.lang.String timestampAttribute, java.lang.String idAttribute, int numShards)
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, java.lang.String timestampAttribute, java.lang.String idAttribute, int numShards, int publishBatchSize, int publishBatchBytes)
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, java.lang.String timestampAttribute, java.lang.String idAttribute, int numShards, int publishBatchSize, int publishBatchBytes, java.lang.String pubsubRootUrl)
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, java.lang.String timestampAttribute, java.lang.String idAttribute, int numShards, java.lang.String pubsubRootUrl)

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<PubsubMessage> input) Override this method to specify how this PTransform should be expanded on the given InputT.
@Nullable java.lang.String	getIdAttribute() Get the id attribute.
@Nullable java.lang.String	getTimestampAttribute() Get the timestamp attribute.
@Nullable PubsubClient.TopicPath	getTopic() Get the topic being written to.
@Nullable ValueProvider<PubsubClient.TopicPath>	getTopicProvider() Get the ValueProvider for the topic being written to.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
                           ValueProvider<PubsubClient.TopicPath> topic,
                           java.lang.String timestampAttribute,
                           java.lang.String idAttribute,
                           int numShards)

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
                           ValueProvider<PubsubClient.TopicPath> topic,
                           java.lang.String timestampAttribute,
                           java.lang.String idAttribute,
                           int numShards,
                           java.lang.String pubsubRootUrl)

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
                           ValueProvider<PubsubClient.TopicPath> topic,
                           java.lang.String timestampAttribute,
                           java.lang.String idAttribute,
                           int numShards,
                           int publishBatchSize,
                           int publishBatchBytes)

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
                           ValueProvider<PubsubClient.TopicPath> topic,
                           java.lang.String timestampAttribute,
                           java.lang.String idAttribute,
                           int numShards,
                           int publishBatchSize,
                           int publishBatchBytes,
                           java.lang.String pubsubRootUrl)

Method Detail

getTopic

public @Nullable PubsubClient.TopicPath getTopic()

Get the topic being written to.

getTopicProvider

public @Nullable ValueProvider<PubsubClient.TopicPath> getTopicProvider()

Get the ValueProvider for the topic being written to.

getTimestampAttribute

public @Nullable java.lang.String getTimestampAttribute()

Get the timestamp attribute.

getIdAttribute

public @Nullable java.lang.String getIdAttribute()

Get the id attribute.

expand

public PDone expand(PCollection<PubsubMessage> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<PubsubMessage>,PDone>