Package org.apache.beam.sdk.io.aws2.kinesis

Class KinesisIO.Read

java.lang.Object
org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<KinesisRecord>>
org.apache.beam.sdk.io.aws2.kinesis.KinesisIO.Read
All Implemented Interfaces:
Serializable, HasDisplayData
Enclosing class:
KinesisIO
public abstract static class KinesisIO.Read extends PTransform<PBegin,PCollection<KinesisRecord>>
Implementation of KinesisIO.read().
See Also:

  • Constructor Details

    • Read

      public Read()

  • Method Details

    • withStreamName

      public KinesisIO.Read withStreamName(String streamName)
      Specify reading from streamName.

    • withConsumerArn

      public KinesisIO.Read withConsumerArn(String consumerArn)
      Specify consumer ARN to enable Enhanced Fan-Out.

    • withInitialPositionInStream

      public KinesisIO.Read withInitialPositionInStream(software.amazon.kinesis.common.InitialPositionInStream initialPosition)
      Specify reading from some initial position in stream.

    • withInitialTimestampInStream

      public KinesisIO.Read withInitialTimestampInStream(Instant initialTimestamp)
      Specify reading beginning at given Instant. This Instant must be in the past, i.e. before Instant.now().

    • withClientConfiguration

      public KinesisIO.Read withClientConfiguration(ClientConfiguration config)
      Configuration of Kinesis invalid input: '&' Cloudwatch clients.

    • withMaxNumRecords

      public KinesisIO.Read withMaxNumRecords(long maxNumRecords)
      Specifies to read at most a given number of records.

    • withMaxReadTime

      public KinesisIO.Read withMaxReadTime(Duration maxReadTime)
      Specifies to read records during maxReadTime.

    • withUpToDateThreshold

      public KinesisIO.Read withUpToDateThreshold(Duration upToDateThreshold)
      Specifies how late records consumed by this source can be to still be considered on time. When this limit is exceeded the actual backlog size will be evaluated and the runner might decide to scale the amount of resources allocated to the pipeline in order to speed up ingestion.

    • withRequestRecordsLimit

      public KinesisIO.Read withRequestRecordsLimit(int limit)
      Specifies the maximum number of records in GetRecordsResult returned by GetRecords call which is limited by 10K records. If should be adjusted according to average size of data record to prevent shard overloading. More details can be found here: API_GetRecords

    • withArrivalTimeWatermarkPolicy

      public KinesisIO.Read withArrivalTimeWatermarkPolicy()
      Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

    • withArrivalTimeWatermarkPolicy

      public KinesisIO.Read withArrivalTimeWatermarkPolicy(Duration watermarkIdleDurationThreshold)
      Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

      Denotes the duration for which the watermark can be idle.

    • withProcessingTimeWatermarkPolicy

      public KinesisIO.Read withProcessingTimeWatermarkPolicy()
      Specifies the WatermarkPolicyFactory as ProcessingTimeWatermarkPolicyFactory.

    • withCustomWatermarkPolicy

      public KinesisIO.Read withCustomWatermarkPolicy(WatermarkPolicyFactory watermarkPolicyFactory)
      Specifies the WatermarkPolicyFactory as a custom watermarkPolicyFactory.
      Parameters:
      watermarkPolicyFactory - Custom Watermark policy factory.

    • withFixedDelayRateLimitPolicy

      public KinesisIO.Read withFixedDelayRateLimitPolicy()
      Specifies a fixed delay rate limit policy with the default delay of 1 second.

    • withFixedDelayRateLimitPolicy

      public KinesisIO.Read withFixedDelayRateLimitPolicy(Duration delay)
      Specifies a fixed delay rate limit policy with the given delay.
      Parameters:
      delay - Denotes the fixed delay duration.

    • withDynamicDelayRateLimitPolicy

      public KinesisIO.Read withDynamicDelayRateLimitPolicy(Supplier<Duration> delay)
      Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value. This can be used to change the polling interval of a running pipeline based on some external configuration source, for example.
      Parameters:
      delay - The function to invoke to get the next delay duration.

    • withCustomRateLimitPolicy

      public KinesisIO.Read withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory)
      Specifies the RateLimitPolicyFactory for a custom rate limiter.
      Parameters:
      rateLimitPolicyFactory - Custom rate limit policy factory.

    • withMaxCapacityPerShard

      public KinesisIO.Read withMaxCapacityPerShard(Integer maxCapacity)
      Specifies the maximum number of messages per one shard.

      Note: When using consumers with dedicated throughput (Enhanced Fan-Out), this capacity corresponds to the number of in-flight shard events which itself can contain multiple, potentially even aggregated records.

      See Also:

    • expand

      public PCollection<KinesisRecord> expand(PBegin input)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PBegin,PCollection<KinesisRecord>>