public class PubsubIO
extends java.lang.Object
PTransforms for Cloud Pub/Sub streams. These transforms create and consume
unbounded PCollections.
In order to use local emulator for Pubsub you should use PubsubOptions#setPubsubRootUrl(String) method to set host and port of your local emulator.
Permission requirements depend on the PipelineRunner that is used to execute the Beam
pipeline. Please refer to the documentation of corresponding PipelineRunners for more details.
| Modifier and Type | Class and Description |
|---|---|
static class |
PubsubIO.PubsubSubscription
Class representing a Cloud Pub/Sub Subscription.
|
static class |
PubsubIO.PubsubTopic
Class representing a Cloud Pub/Sub Topic.
|
static class |
PubsubIO.Read<T>
Implementation of
read(). |
static class |
PubsubIO.Write<T>
Implementation of
write(). |
| Modifier and Type | Method and Description |
|---|---|
static PubsubIO.Read<GenericRecord> |
readAvroGenericRecords(Schema avroSchema)
Returns a
PTransform that continuously reads binary encoded Avro messages into the Avro
GenericRecord type. |
static <T> PubsubIO.Read<T> |
readAvros(java.lang.Class<T> clazz)
Returns A
PTransform that continuously reads binary encoded Avro messages of the given
type from a Google Cloud Pub/Sub stream. |
static <T> PubsubIO.Read<T> |
readAvrosWithBeamSchema(java.lang.Class<T> clazz)
Returns a
PTransform that continuously reads binary encoded Avro messages of the
specific type. |
static PubsubIO.Read<PubsubMessage> |
readMessages()
Returns A
PTransform that continuously reads from a Google Cloud Pub/Sub stream. |
static PubsubIO.Read<PubsubMessage> |
readMessagesWithAttributes()
Returns A
PTransform that continuously reads from a Google Cloud Pub/Sub stream. |
static PubsubIO.Read<PubsubMessage> |
readMessagesWithAttributesAndMessageId()
Returns A
PTransform that continuously reads from a Google Cloud Pub/Sub stream. |
static <T> PubsubIO.Read<T> |
readMessagesWithCoderAndParseFn(Coder<T> coder,
SimpleFunction<PubsubMessage,T> parseFn)
Returns A
PTransform that continuously reads from a Google Cloud Pub/Sub stream,
mapping each PubsubMessage into type T using the supplied parse function and coder. |
static PubsubIO.Read<PubsubMessage> |
readMessagesWithMessageId()
Returns A
PTransform that continuously reads from a Google Cloud Pub/Sub stream. |
static <T extends Message> |
readProtos(java.lang.Class<T> messageClass)
Returns A
PTransform that continuously reads binary encoded protobuf messages of the
given type from a Google Cloud Pub/Sub stream. |
static PubsubIO.Read<java.lang.String> |
readStrings()
Returns A
PTransform that continuously reads UTF-8 encoded strings from a Google Cloud
Pub/Sub stream. |
static <T> PubsubIO.Write<T> |
writeAvros(java.lang.Class<T> clazz)
Returns A
PTransform that writes binary encoded Avro messages of a given type to a
Google Cloud Pub/Sub stream. |
static PubsubIO.Write<PubsubMessage> |
writeMessages()
Returns A
PTransform that writes to a Google Cloud Pub/Sub stream. |
static <T extends Message> |
writeProtos(java.lang.Class<T> messageClass)
Returns A
PTransform that writes binary encoded protobuf messages of a given type to a
Google Cloud Pub/Sub stream. |
static PubsubIO.Write<java.lang.String> |
writeStrings()
Returns A
PTransform that writes UTF-8 encoded strings to a Google Cloud Pub/Sub
stream. |
public static PubsubIO.Read<PubsubMessage> readMessages()
PTransform that continuously reads from a Google Cloud Pub/Sub stream. The
messages will only contain a payload, but no attributes.public static PubsubIO.Read<PubsubMessage> readMessagesWithMessageId()
PTransform that continuously reads from a Google Cloud Pub/Sub stream. The
messages will only contain a payload with the messageId from PubSub, but no attributes.public static PubsubIO.Read<PubsubMessage> readMessagesWithAttributes()
PTransform that continuously reads from a Google Cloud Pub/Sub stream. The
messages will contain both a payload and attributes.public static PubsubIO.Read<PubsubMessage> readMessagesWithAttributesAndMessageId()
PTransform that continuously reads from a Google Cloud Pub/Sub stream. The
messages will contain both a payload and attributes, along with the messageId from PubSub.public static PubsubIO.Read<java.lang.String> readStrings()
PTransform that continuously reads UTF-8 encoded strings from a Google Cloud
Pub/Sub stream.public static <T extends Message> PubsubIO.Read<T> readProtos(java.lang.Class<T> messageClass)
PTransform that continuously reads binary encoded protobuf messages of the
given type from a Google Cloud Pub/Sub stream.public static <T> PubsubIO.Read<T> readAvros(java.lang.Class<T> clazz)
PTransform that continuously reads binary encoded Avro messages of the given
type from a Google Cloud Pub/Sub stream.public static <T> PubsubIO.Read<T> readMessagesWithCoderAndParseFn(Coder<T> coder, SimpleFunction<PubsubMessage,T> parseFn)
PTransform that continuously reads from a Google Cloud Pub/Sub stream,
mapping each PubsubMessage into type T using the supplied parse function and coder.@Experimental(value=SCHEMAS) public static PubsubIO.Read<GenericRecord> readAvroGenericRecords(Schema avroSchema)
PTransform that continuously reads binary encoded Avro messages into the Avro
GenericRecord type.
Beam will infer a schema for the Avro schema. This allows the output to be used by SQL and by the schema-transform library.
@Experimental(value=SCHEMAS) public static <T> PubsubIO.Read<T> readAvrosWithBeamSchema(java.lang.Class<T> clazz)
PTransform that continuously reads binary encoded Avro messages of the
specific type.
Beam will infer a schema for the Avro schema. This allows the output to be used by SQL and by the schema-transform library.
public static PubsubIO.Write<PubsubMessage> writeMessages()
PTransform that writes to a Google Cloud Pub/Sub stream.public static PubsubIO.Write<java.lang.String> writeStrings()
PTransform that writes UTF-8 encoded strings to a Google Cloud Pub/Sub
stream.public static <T extends Message> PubsubIO.Write<T> writeProtos(java.lang.Class<T> messageClass)
PTransform that writes binary encoded protobuf messages of a given type to a
Google Cloud Pub/Sub stream.public static <T> PubsubIO.Write<T> writeAvros(java.lang.Class<T> clazz)
PTransform that writes binary encoded Avro messages of a given type to a
Google Cloud Pub/Sub stream.