org.apache.beam.sdk.io.aws2.kinesis

Class KinesisIO.Read

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<KinesisRecord>>

org.apache.beam.sdk.io.aws2.kinesis.KinesisIO.Read

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

KinesisIO

public abstract static class KinesisIO.Read
extends PTransform<PBegin,PCollection<KinesisRecord>>

Implementation of KinesisIO.read().

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Read()

Method Summary

Modifier and Type	Method and Description
PCollection<KinesisRecord>	expand(PBegin input) Override this method to specify how this PTransform should be expanded on the given InputT.
KinesisIO.Read	withArrivalTimeWatermarkPolicy() Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.
KinesisIO.Read	withArrivalTimeWatermarkPolicy(Duration watermarkIdleDurationThreshold) Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.
KinesisIO.Read	withClientConfiguration(ClientConfiguration config) Configuration of Kinesis & Cloudwatch clients.
KinesisIO.Read	withConsumerArn(java.lang.String consumerArn) Specify consumer ARN to enable Enhanced Fan-Out.
KinesisIO.Read	withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory) Specifies the RateLimitPolicyFactory for a custom rate limiter.
KinesisIO.Read	withCustomWatermarkPolicy(WatermarkPolicyFactory watermarkPolicyFactory) Specifies the WatermarkPolicyFactory as a custom watermarkPolicyFactory.
KinesisIO.Read	withDynamicDelayRateLimitPolicy(java.util.function.Supplier<Duration> delay) Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value.
KinesisIO.Read	withFixedDelayRateLimitPolicy() Specifies a fixed delay rate limit policy with the default delay of 1 second.
KinesisIO.Read	withFixedDelayRateLimitPolicy(Duration delay) Specifies a fixed delay rate limit policy with the given delay.
KinesisIO.Read	withInitialPositionInStream(software.amazon.kinesis.common.InitialPositionInStream initialPosition) Specify reading from some initial position in stream.
KinesisIO.Read	withInitialTimestampInStream(Instant initialTimestamp) Specify reading beginning at given Instant.
KinesisIO.Read	withMaxCapacityPerShard(java.lang.Integer maxCapacity) Specifies the maximum number of messages per one shard.
KinesisIO.Read	withMaxNumRecords(long maxNumRecords) Specifies to read at most a given number of records.
KinesisIO.Read	withMaxReadTime(Duration maxReadTime) Specifies to read records during maxReadTime.
KinesisIO.Read	withProcessingTimeWatermarkPolicy() Specifies the WatermarkPolicyFactory as ProcessingTimeWatermarkPolicyFactory.
KinesisIO.Read	withRequestRecordsLimit(int limit) Specifies the maximum number of records in GetRecordsResult returned by GetRecords call which is limited by 10K records.
KinesisIO.Read	withStreamName(java.lang.String streamName) Specify reading from streamName.
KinesisIO.Read	withUpToDateThreshold(Duration upToDateThreshold) Specifies how late records consumed by this source can be to still be considered on time.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Read

public Read()

Method Detail

withStreamName

public KinesisIO.Read withStreamName(java.lang.String streamName)

Specify reading from streamName.

withConsumerArn

public KinesisIO.Read withConsumerArn(java.lang.String consumerArn)

Specify consumer ARN to enable Enhanced Fan-Out.

withInitialPositionInStream

public KinesisIO.Read withInitialPositionInStream(software.amazon.kinesis.common.InitialPositionInStream initialPosition)

Specify reading from some initial position in stream.

withInitialTimestampInStream

public KinesisIO.Read withInitialTimestampInStream(Instant initialTimestamp)

Specify reading beginning at given Instant. This Instant must be in the past, i.e. before Instant.now().

withClientConfiguration

public KinesisIO.Read withClientConfiguration(ClientConfiguration config)

Configuration of Kinesis & Cloudwatch clients.

withMaxNumRecords

public KinesisIO.Read withMaxNumRecords(long maxNumRecords)

Specifies to read at most a given number of records.

withMaxReadTime

public KinesisIO.Read withMaxReadTime(Duration maxReadTime)

Specifies to read records during maxReadTime.

withUpToDateThreshold

public KinesisIO.Read withUpToDateThreshold(Duration upToDateThreshold)

Specifies how late records consumed by this source can be to still be considered on time. When this limit is exceeded the actual backlog size will be evaluated and the runner might decide to scale the amount of resources allocated to the pipeline in order to speed up ingestion.

withRequestRecordsLimit

public KinesisIO.Read withRequestRecordsLimit(int limit)

Specifies the maximum number of records in GetRecordsResult returned by GetRecords call which is limited by 10K records. If should be adjusted according to average size of data record to prevent shard overloading. More details can be found here: API_GetRecords

withArrivalTimeWatermarkPolicy

public KinesisIO.Read withArrivalTimeWatermarkPolicy()

Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

withArrivalTimeWatermarkPolicy

public KinesisIO.Read withArrivalTimeWatermarkPolicy(Duration watermarkIdleDurationThreshold)

Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

Denotes the duration for which the watermark can be idle.

withProcessingTimeWatermarkPolicy

public KinesisIO.Read withProcessingTimeWatermarkPolicy()

Specifies the WatermarkPolicyFactory as ProcessingTimeWatermarkPolicyFactory.

withCustomWatermarkPolicy

public KinesisIO.Read withCustomWatermarkPolicy(WatermarkPolicyFactory watermarkPolicyFactory)

Specifies the WatermarkPolicyFactory as a custom watermarkPolicyFactory.

Parameters:

watermarkPolicyFactory - Custom Watermark policy factory.

withFixedDelayRateLimitPolicy

public KinesisIO.Read withFixedDelayRateLimitPolicy()

Specifies a fixed delay rate limit policy with the default delay of 1 second.

withFixedDelayRateLimitPolicy

public KinesisIO.Read withFixedDelayRateLimitPolicy(Duration delay)

Specifies a fixed delay rate limit policy with the given delay.

Parameters:

delay - Denotes the fixed delay duration.

withDynamicDelayRateLimitPolicy

public KinesisIO.Read withDynamicDelayRateLimitPolicy(java.util.function.Supplier<Duration> delay)

Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value. This can be used to change the polling interval of a running pipeline based on some external configuration source, for example.

Parameters:

delay - The function to invoke to get the next delay duration.

withCustomRateLimitPolicy

public KinesisIO.Read withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory)

Specifies the RateLimitPolicyFactory for a custom rate limiter.

Parameters:

rateLimitPolicyFactory - Custom rate limit policy factory.

withMaxCapacityPerShard

public KinesisIO.Read withMaxCapacityPerShard(java.lang.Integer maxCapacity)

Specifies the maximum number of messages per one shard.

Note: When using consumers with dedicated throughput (Enhanced Fan-Out), this capacity corresponds to the number of in-flight shard events which itself can contain multiple, potentially even aggregated records.

See Also:

#withConsumerArn(String)}

expand

public PCollection<KinesisRecord> expand(PBegin input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PBegin,PCollection<KinesisRecord>>