org.apache.beam.sdk.io.kinesis

Class KinesisIO.Read<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<T>>

org.apache.beam.sdk.io.kinesis.KinesisIO.Read<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

KinesisIO

public abstract static class KinesisIO.Read<T>
extends PTransform<PBegin,PCollection<T>>

Implementation of KinesisIO.read().

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Read()

Method Summary

Modifier and Type	Method and Description
PCollection<T>	expand(PBegin input) Override this method to specify how this PTransform should be expanded on the given InputT.
KinesisIO.Read<T>	withArrivalTimeWatermarkPolicy() Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.
KinesisIO.Read<T>	withArrivalTimeWatermarkPolicy(Duration watermarkIdleDurationThreshold) Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.
KinesisIO.Read<T>	withAWSClientsProvider(AWSClientsProvider awsClientsProvider) Allows to specify custom AWSClientsProvider.
KinesisIO.Read<T>	withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider, Regions region) Specify AWSCredentialsProvider and region to be used to read from Kinesis.
KinesisIO.Read<T>	withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider, Regions region, java.lang.String serviceEndpoint) Specify AWSCredentialsProvider and region to be used to read from Kinesis.
KinesisIO.Read<T>	withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider, Regions region, java.lang.String serviceEndpoint, boolean verifyCertificate) Specify AWSCredentialsProvider and region to be used to read from Kinesis.
KinesisIO.Read<T>	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, Regions region) Specify credential details and region to be used to read from Kinesis.
KinesisIO.Read<T>	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, Regions region, java.lang.String serviceEndpoint) Specify credential details and region to be used to read from Kinesis.
KinesisIO.Read<T>	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, Regions region, java.lang.String serviceEndpoint, boolean verifyCertificate) Specify credential details and region to be used to read from Kinesis.
KinesisIO.Read<T>	withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory) Specifies the RateLimitPolicyFactory for a custom rate limiter.
KinesisIO.Read<T>	withCustomWatermarkPolicy(WatermarkPolicyFactory watermarkPolicyFactory) Specifies the WatermarkPolicyFactory as a custom watermarkPolicyFactory.
KinesisIO.Read<T>	withDynamicDelayRateLimitPolicy(java.util.function.Supplier<Duration> delay) Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value.
KinesisIO.Read<T>	withFixedDelayRateLimitPolicy() Specifies a fixed delay rate limit policy with the default delay of 1 second.
KinesisIO.Read<T>	withFixedDelayRateLimitPolicy(Duration delay) Specifies a fixed delay rate limit policy with the given delay.
KinesisIO.Read<T>	withInitialPositionInStream(com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPosition) Specify reading from some initial position in stream.
KinesisIO.Read<T>	withInitialTimestampInStream(Instant initialTimestamp) Specify reading beginning at given Instant.
KinesisIO.Read<T>	withMaxCapacityPerShard(java.lang.Integer maxCapacity) Specifies the maximum number of messages per one shard.
KinesisIO.Read<T>	withMaxNumRecords(long maxNumRecords) Specifies to read at most a given number of records.
KinesisIO.Read<T>	withMaxReadTime(Duration maxReadTime) Specifies to read records during maxReadTime.
KinesisIO.Read<T>	withProcessingTimeWatermarkPolicy() Specifies the WatermarkPolicyFactory as ProcessingTimeWatermarkPolicyFactory.
KinesisIO.Read<T>	withRequestRecordsLimit(int limit) Specifies the maximum number of records in GetRecordsResult returned by GetRecords call which is limited by 10K records.
KinesisIO.Read<T>	withStreamName(java.lang.String streamName) Specify reading from streamName.
KinesisIO.Read<T>	withUpToDateThreshold(Duration upToDateThreshold) Specifies how late records consumed by this source can be to still be considered on time.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Read

public Read()

Method Detail

withStreamName

public KinesisIO.Read<T> withStreamName(java.lang.String streamName)

Specify reading from streamName.

withInitialPositionInStream

public KinesisIO.Read<T> withInitialPositionInStream(com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPosition)

Specify reading from some initial position in stream.

withInitialTimestampInStream

public KinesisIO.Read<T> withInitialTimestampInStream(Instant initialTimestamp)

Specify reading beginning at given Instant. This Instant must be in the past, i.e. before Instant.now().

withAWSClientsProvider

public KinesisIO.Read<T> withAWSClientsProvider(AWSClientsProvider awsClientsProvider)

Allows to specify custom AWSClientsProvider. AWSClientsProvider provides AmazonKinesis and AmazonCloudWatch instances which are later used for communication with Kinesis. You should use this method if withAWSClientsProvider(AWSCredentialsProvider, Regions) does not suit your needs.

withAWSClientsProvider

public KinesisIO.Read<T> withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider,
                                                Regions region)

Specify AWSCredentialsProvider and region to be used to read from Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

withAWSClientsProvider

public KinesisIO.Read<T> withAWSClientsProvider(java.lang.String awsAccessKey,
                                                java.lang.String awsSecretKey,
                                                Regions region)

Specify credential details and region to be used to read from Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

withAWSClientsProvider

public KinesisIO.Read<T> withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider,
                                                Regions region,
                                                java.lang.String serviceEndpoint)

Specify AWSCredentialsProvider and region to be used to read from Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with a kinesis service emulator.

withAWSClientsProvider

public KinesisIO.Read<T> withAWSClientsProvider(java.lang.String awsAccessKey,
                                                java.lang.String awsSecretKey,
                                                Regions region,
                                                java.lang.String serviceEndpoint)

Specify credential details and region to be used to read from Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with a kinesis service emulator.

withAWSClientsProvider

public KinesisIO.Read<T> withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider,
                                                Regions region,
                                                java.lang.String serviceEndpoint,
                                                boolean verifyCertificate)

Specify AWSCredentialsProvider and region to be used to read from Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

The verifyCertificate disables or enables certificate verification. Never set it to false in production.

withAWSClientsProvider

public KinesisIO.Read<T> withAWSClientsProvider(java.lang.String awsAccessKey,
                                                java.lang.String awsSecretKey,
                                                Regions region,
                                                java.lang.String serviceEndpoint,
                                                boolean verifyCertificate)

Specify credential details and region to be used to read from Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

The verifyCertificate disables or enables certificate verification. Never set it to false in production.

withMaxNumRecords

public KinesisIO.Read<T> withMaxNumRecords(long maxNumRecords)

Specifies to read at most a given number of records.

withMaxReadTime

public KinesisIO.Read<T> withMaxReadTime(Duration maxReadTime)

Specifies to read records during maxReadTime.

withUpToDateThreshold

public KinesisIO.Read<T> withUpToDateThreshold(Duration upToDateThreshold)

Specifies how late records consumed by this source can be to still be considered on time. When this limit is exceeded the actual backlog size will be evaluated and the runner might decide to scale the amount of resources allocated to the pipeline in order to speed up ingestion.

withRequestRecordsLimit

public KinesisIO.Read<T> withRequestRecordsLimit(int limit)

Specifies the maximum number of records in GetRecordsResult returned by GetRecords call which is limited by 10K records. If should be adjusted according to average size of data record to prevent shard overloading. More details can be found here: API_GetRecords

withArrivalTimeWatermarkPolicy

public KinesisIO.Read<T> withArrivalTimeWatermarkPolicy()

Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

withArrivalTimeWatermarkPolicy

public KinesisIO.Read<T> withArrivalTimeWatermarkPolicy(Duration watermarkIdleDurationThreshold)

Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

Denotes the duration for which the watermark can be idle.

withProcessingTimeWatermarkPolicy

public KinesisIO.Read<T> withProcessingTimeWatermarkPolicy()

Specifies the WatermarkPolicyFactory as ProcessingTimeWatermarkPolicyFactory.

withCustomWatermarkPolicy

public KinesisIO.Read<T> withCustomWatermarkPolicy(WatermarkPolicyFactory watermarkPolicyFactory)

Specifies the WatermarkPolicyFactory as a custom watermarkPolicyFactory.

Parameters:

watermarkPolicyFactory - Custom Watermark policy factory.

withFixedDelayRateLimitPolicy

public KinesisIO.Read<T> withFixedDelayRateLimitPolicy()

Specifies a fixed delay rate limit policy with the default delay of 1 second.

withFixedDelayRateLimitPolicy

public KinesisIO.Read<T> withFixedDelayRateLimitPolicy(Duration delay)

Specifies a fixed delay rate limit policy with the given delay.

Parameters:

delay - Denotes the fixed delay duration.

withDynamicDelayRateLimitPolicy

public KinesisIO.Read<T> withDynamicDelayRateLimitPolicy(java.util.function.Supplier<Duration> delay)

Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value. This can be used to change the polling interval of a running pipeline based on some external configuration source, for example.

Parameters:

delay - The function to invoke to get the next delay duration.

withCustomRateLimitPolicy

public KinesisIO.Read<T> withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory)

Specifies the RateLimitPolicyFactory for a custom rate limiter.

Parameters:

rateLimitPolicyFactory - Custom rate limit policy factory.

withMaxCapacityPerShard

public KinesisIO.Read<T> withMaxCapacityPerShard(java.lang.Integer maxCapacity)

Specifies the maximum number of messages per one shard.

expand

public PCollection<T> expand(PBegin input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PBegin,PCollection<T>>