Package org.apache.beam.sdk.io.gcp.pubsub

Class PubsubReadSchemaTransformConfiguration

java.lang.Object
org.apache.beam.sdk.io.gcp.pubsub.PubsubReadSchemaTransformConfiguration
@DefaultSchema(AutoValueSchema.class) public abstract class PubsubReadSchemaTransformConfiguration extends Object
Configuration for reading from Pub/Sub.

Internal only: This class is actively being worked on, and it will likely change. We provide no backwards compatibility guarantees, and it should not be implemented outside the Beam repository.

  • Constructor Details

    • PubsubReadSchemaTransformConfiguration

      public PubsubReadSchemaTransformConfiguration()

  • Method Details

    • getTopic

      @SchemaFieldDescription("The name of the topic to consume data from. If a topic is specified, will create a new subscription for that topic and start consuming from that point. Either a topic or a subscription must be provided. Format: projects/${PROJECT}/topics/${TOPIC}") @Nullable public abstract String getTopic()

    • getSubscription

      @SchemaFieldDescription("The name of the subscription to consume data. Either a topic or subscription must be provided. Format: projects/${PROJECT}/subscriptions/${SUBSCRIPTION}") @Nullable public abstract String getSubscription()

    • getFormat

      @SchemaFieldDescription("The encoding format for the data stored in Pubsub. Valid options are: RAW,AVRO,JSON") public abstract String getFormat()

    • getSchema

      @SchemaFieldDescription("The schema in which the data is encoded in the Pubsub topic. For AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/).") public abstract String getSchema()

    • getAttributes

      @SchemaFieldDescription("Any additional pubsub attributes that should be populated as String fields in the ouptut rows.") @Nullable public abstract List<String> getAttributes()

    • getAttributesMap

      @SchemaFieldDescription("Any additional field that should be populated with the full set of PubSub attributes.") @Nullable public abstract String getAttributesMap()

    • getIdAttribute

      @SchemaFieldDescription("When reading from Cloud Pub/Sub where unique record identifiers are provided as Pub/Sub message attributes, specifies the name of the attribute containing the unique identifier. The value of the attribute can be any string that uniquely identifies this record. Pub/Sub cannot guarantee that no duplicate data will be delivered on the Pub/Sub stream. If idAttribute is not provided, Beam cannot guarantee that no duplicate data will be delivered, and deduplication of the stream will be strictly best effort.") @Nullable public abstract String getIdAttribute()

    • getTimestampAttribute

      @SchemaFieldDescription("Specifies the name of the attribute that contains the timestamp, if any. The timestamp value is expected to be represented in the attribute as either (1) a numerical value representing the number of milliseconds since the Unix epoch. For example, if using the Joda time classes, Instant.getMillis() returns the correct value for this attribute. or (2) a String in RFC 3339 format. For example, 2015-10-29T23:41:41.123Z. The sub-second component of the timestamp is optional, and digits beyond the first three (i.e., time units smaller than milliseconds) will be ignored.") @Nullable public abstract String getTimestampAttribute()

    • getErrorHandling

      @SchemaFieldDescription("Specifies how to handle errors.") @Nullable public abstract PubsubReadSchemaTransformConfiguration.ErrorHandling getErrorHandling()

    • getClientFactory

      @Nullable public abstract PubsubTestClient.PubsubTestClientFactory getClientFactory()

    • getClock

      @Nullable public abstract com.google.api.client.util.Clock getClock()

    • builder

      public static PubsubReadSchemaTransformConfiguration.Builder builder()