Package org.apache.beam.sdk.io.gcp.bigtable

Class BigtableIO.ReadChangeStream

java.lang.Object
org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<KV<ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>>>
org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.ReadChangeStream
All Implemented Interfaces:
Serializable, HasDisplayData
Enclosing class:
BigtableIO
public abstract static class BigtableIO.ReadChangeStream extends PTransform<PBegin,PCollection<KV<ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>>>
See Also:

  • Constructor Details

    • ReadChangeStream

      public ReadChangeStream()

  • Method Details

    • withProjectId

      public BigtableIO.ReadChangeStream withProjectId(String projectId)
      Returns a new BigtableIO.ReadChangeStream that will stream from the Cloud Bigtable project indicated by given parameter, requires withInstanceId(java.lang.String) to be called to determine the instance.

      Does not modify this object.

    • withInstanceId

      public BigtableIO.ReadChangeStream withInstanceId(String instanceId)
      Returns a new BigtableIO.ReadChangeStream that will stream from the Cloud Bigtable instance indicated by given parameter, requires withProjectId(java.lang.String) to be called to determine the project.

      Does not modify this object.

    • withTableId

      public BigtableIO.ReadChangeStream withTableId(String tableId)
      Returns a new BigtableIO.ReadChangeStream that will stream from the specified table.

      Does not modify this object.

    • withAppProfileId

      public BigtableIO.ReadChangeStream withAppProfileId(String appProfileId)
      Returns a new BigtableIO.ReadChangeStream that will stream from the cluster specified by app profile id.

      This must use single-cluster routing policy. If not setting a separate app profile for the metadata table with withMetadataTableAppProfileId(java.lang.String), this app profile also needs to enable allow single-row transactions.

      Does not modify this object.

    • withStartTime

      public BigtableIO.ReadChangeStream withStartTime(Instant startTime)
      Returns a new BigtableIO.ReadChangeStream that will start streaming at the specified start time.

      Does not modify this object.

    • withChangeStreamName

      public BigtableIO.ReadChangeStream withChangeStreamName(String changeStreamName)
      Returns a new BigtableIO.ReadChangeStream that uses changeStreamName as prefix for the metadata table.

      Does not modify this object.

    • withExistingPipelineOptions

      public BigtableIO.ReadChangeStream withExistingPipelineOptions(BigtableIO.ExistingPipelineOptions existingPipelineOptions)
      Returns a new BigtableIO.ReadChangeStream that decides what to do if an existing pipeline exists with the same change stream name.

      Does not modify this object.

    • withMetadataTableProjectId

      public BigtableIO.ReadChangeStream withMetadataTableProjectId(String projectId)
      Returns a new BigtableIO.ReadChangeStream that will use the Cloud Bigtable project indicated by given parameter to manage the metadata of the stream.

      Optional: defaults to value from withProjectId

      Does not modify this object.

    • withMetadataTableInstanceId

      public BigtableIO.ReadChangeStream withMetadataTableInstanceId(String instanceId)
      Returns a new BigtableIO.ReadChangeStream that will use the Cloud Bigtable instance indicated by given parameter to manage the metadata of the stream.

      Optional: defaults to value from withInstanceId

      Does not modify this object.

    • withMetadataTableTableId

      public BigtableIO.ReadChangeStream withMetadataTableTableId(String tableId)
      Returns a new BigtableIO.ReadChangeStream that will use specified table to store the metadata of the stream.

      Optional: defaults to value from withTableId

      Does not modify this object.

    • withMetadataTableAppProfileId

      public BigtableIO.ReadChangeStream withMetadataTableAppProfileId(String appProfileId)
      Returns a new BigtableIO.ReadChangeStream that will use the cluster specified by app profile id to store the metadata of the stream.

      Optional: defaults to value from withAppProfileId

      This must use single-cluster routing policy with allow single-row transactions enabled.

      Does not modify this object.

    • withCreateOrUpdateMetadataTable

      public BigtableIO.ReadChangeStream withCreateOrUpdateMetadataTable(boolean shouldCreate)
      Returns a new BigtableIO.ReadChangeStream that, if set to true, will create or update metadata table before launching pipeline. Otherwise, it is expected that a metadata table with correct schema exists.

      Optional: defaults to true

      Does not modify this object.

    • withBacklogReplicationAdjustment

      public BigtableIO.ReadChangeStream withBacklogReplicationAdjustment(Duration adjustment)
      Returns a new BigtableIO.ReadChangeStream that overrides the replication delay adjustment duration with the provided duration.

      Backlog is calculated for each partition using watermarkLag * throughput. Replication delay holds back the watermark for each partition. This can cause the backlog to stay persistently above dataflow's downscaling threshold (10 seconds) even when a pipeline is caught up.

      This adjusts the backlog downward to account for this. For unreplicated instances it can be set to zero to upscale as quickly as possible.

      Optional: defaults to 30 seconds.

      Does not modify this object.

    • withReadChangeStreamTimeout

      public BigtableIO.ReadChangeStream withReadChangeStreamTimeout(Duration timeout)
      Returns a new BigtableIO.ReadChangeStream that overrides timeout for ReadChangeStream requests.

      This is useful to override the default of 15s timeout if the checkpoint duration is longer than 15s. Setting this value to longer (to add some padding) than periodic checkpoint duration ensures that ReadChangeStream will stream until the next checkpoint is initiated.

      Optional: defaults to 15 seconds.

      Does not modify this object.

    • withoutValidation

      public BigtableIO.ReadChangeStream withoutValidation()
      Disables validation that the table being read and the metadata table exists, and that the app profile used is single cluster and single row transaction enabled. Set this option if the caller does not have additional Bigtable permissions to validate the configurations. NOTE this also disabled creating or updating the metadata table because that also requires additional permissions, essentially setting withCreateOrUpdateMetadataTable(boolean) to false.

    • validate

      public void validate(PipelineOptions options)
      Description copied from class: PTransform
      Called before running the Pipeline to verify this transform is fully and correctly specified.

      By default, does nothing.

      Overrides:
      validate in class PTransform<PBegin,PCollection<KV<ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>>>

    • expand

      public PCollection<KV<ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>> expand(PBegin input)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PBegin,PCollection<KV<ByteString,com.google.cloud.bigtable.data.v2.models.ChangeStreamMutation>>>