org.apache.beam.sdk.io.kinesis

Class KinesisIO.Read

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<KinesisRecord>>

org.apache.beam.sdk.io.kinesis.KinesisIO.Read

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

KinesisIO

public abstract static class KinesisIO.Read
extends PTransform<PBegin,PCollection<KinesisRecord>>

Implementation of KinesisIO.read().

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Constructor Summary

Constructors

Constructor and Description
Read()

Method Summary

Modifier and Type	Method and Description
PCollection<KinesisRecord>	expand(PBegin input) Override this method to specify how this PTransform should be expanded on the given InputT.
KinesisIO.Read	withArrivalTimeWatermarkPolicy() Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.
KinesisIO.Read	withArrivalTimeWatermarkPolicy(Duration watermarkIdleDurationThreshold) Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.
KinesisIO.Read	withAWSClientsProvider(AWSClientsProvider awsClientsProvider) Allows to specify custom AWSClientsProvider.
KinesisIO.Read	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, Regions region) Specify credential details and region to be used to read from Kinesis.
KinesisIO.Read	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, Regions region, java.lang.String serviceEndpoint) Specify credential details and region to be used to read from Kinesis.
KinesisIO.Read	withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory) Specifies the RateLimitPolicyFactory for a custom rate limiter.
KinesisIO.Read	withCustomWatermarkPolicy(WatermarkPolicyFactory watermarkPolicyFactory) Specifies the WatermarkPolicyFactory as a custom watermarkPolicyFactory.
KinesisIO.Read	withDynamicDelayRateLimitPolicy(java.util.function.Supplier<Duration> delay) Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value.
KinesisIO.Read	withFixedDelayRateLimitPolicy() Specifies a fixed delay rate limit policy with the default delay of 1 second.
KinesisIO.Read	withFixedDelayRateLimitPolicy(Duration delay) Specifies a fixed delay rate limit policy with the given delay.
KinesisIO.Read	withInitialPositionInStream(com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPosition) Specify reading from some initial position in stream.
KinesisIO.Read	withInitialTimestampInStream(Instant initialTimestamp) Specify reading beginning at given Instant.
KinesisIO.Read	withMaxCapacityPerShard(java.lang.Integer maxCapacity) Specifies the maximum number of messages per one shard.
KinesisIO.Read	withMaxNumRecords(long maxNumRecords) Specifies to read at most a given number of records.
KinesisIO.Read	withMaxReadTime(Duration maxReadTime) Specifies to read records during maxReadTime.
KinesisIO.Read	withProcessingTimeWatermarkPolicy() Specifies the WatermarkPolicyFactory as ProcessingTimeWatermarkPolicyFactory.
KinesisIO.Read	withRequestRecordsLimit(int limit) Specifies the maximum number of records in GetRecordsResult returned by GetRecords call which is limited by 10K records.
KinesisIO.Read	withStreamName(java.lang.String streamName) Specify reading from streamName.
KinesisIO.Read	withUpToDateThreshold(Duration upToDateThreshold) Specifies how late records consumed by this source can be to still be considered on time.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Read

public Read()

Method Detail

withStreamName

public KinesisIO.Read withStreamName(java.lang.String streamName)

Specify reading from streamName.

withInitialPositionInStream

public KinesisIO.Read withInitialPositionInStream(com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream initialPosition)

Specify reading from some initial position in stream.

withInitialTimestampInStream

public KinesisIO.Read withInitialTimestampInStream(Instant initialTimestamp)

Specify reading beginning at given Instant. This Instant must be in the past, i.e. before Instant.now().

withAWSClientsProvider

public KinesisIO.Read withAWSClientsProvider(AWSClientsProvider awsClientsProvider)

Allows to specify custom AWSClientsProvider. AWSClientsProvider provides AmazonKinesis and AmazonCloudWatch instances which are later used for communication with Kinesis. You should use this method if withAWSClientsProvider(String, String, Regions) does not suit your needs.

withAWSClientsProvider

public KinesisIO.Read withAWSClientsProvider(java.lang.String awsAccessKey,
                                             java.lang.String awsSecretKey,
                                             Regions region)

Specify credential details and region to be used to read from Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

withAWSClientsProvider

public KinesisIO.Read withAWSClientsProvider(java.lang.String awsAccessKey,
                                             java.lang.String awsSecretKey,
                                             Regions region,
                                             java.lang.String serviceEndpoint)

Specify credential details and region to be used to read from Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with a kinesis service emulator.

withMaxNumRecords

public KinesisIO.Read withMaxNumRecords(long maxNumRecords)

Specifies to read at most a given number of records.

withMaxReadTime

public KinesisIO.Read withMaxReadTime(Duration maxReadTime)

Specifies to read records during maxReadTime.

withUpToDateThreshold

public KinesisIO.Read withUpToDateThreshold(Duration upToDateThreshold)

Specifies how late records consumed by this source can be to still be considered on time. When this limit is exceeded the actual backlog size will be evaluated and the runner might decide to scale the amount of resources allocated to the pipeline in order to speed up ingestion.

withRequestRecordsLimit

public KinesisIO.Read withRequestRecordsLimit(int limit)

Specifies the maximum number of records in GetRecordsResult returned by GetRecords call which is limited by 10K records. If should be adjusted according to average size of data record to prevent shard overloading. More details can be found here: API_GetRecords

withArrivalTimeWatermarkPolicy

public KinesisIO.Read withArrivalTimeWatermarkPolicy()

Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

withArrivalTimeWatermarkPolicy

public KinesisIO.Read withArrivalTimeWatermarkPolicy(Duration watermarkIdleDurationThreshold)

Specifies the WatermarkPolicyFactory as ArrivalTimeWatermarkPolicyFactory.

Denotes the duration for which the watermark can be idle.

withProcessingTimeWatermarkPolicy

public KinesisIO.Read withProcessingTimeWatermarkPolicy()

Specifies the WatermarkPolicyFactory as ProcessingTimeWatermarkPolicyFactory.

withCustomWatermarkPolicy

public KinesisIO.Read withCustomWatermarkPolicy(WatermarkPolicyFactory watermarkPolicyFactory)

Specifies the WatermarkPolicyFactory as a custom watermarkPolicyFactory.

Parameters:

watermarkPolicyFactory - Custom Watermark policy factory.

withFixedDelayRateLimitPolicy

public KinesisIO.Read withFixedDelayRateLimitPolicy()

Specifies a fixed delay rate limit policy with the default delay of 1 second.

withFixedDelayRateLimitPolicy

public KinesisIO.Read withFixedDelayRateLimitPolicy(Duration delay)

Specifies a fixed delay rate limit policy with the given delay.

Parameters:

delay - Denotes the fixed delay duration.

withDynamicDelayRateLimitPolicy

public KinesisIO.Read withDynamicDelayRateLimitPolicy(java.util.function.Supplier<Duration> delay)

Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value. This can be used to change the polling interval of a running pipeline based on some external configuration source, for example.

Parameters:

delay - The function to invoke to get the next delay duration.

withCustomRateLimitPolicy

public KinesisIO.Read withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory)

Specifies the RateLimitPolicyFactory for a custom rate limiter.

Parameters:

rateLimitPolicyFactory - Custom rate limit policy factory.

withMaxCapacityPerShard

public KinesisIO.Read withMaxCapacityPerShard(java.lang.Integer maxCapacity)

Specifies the maximum number of messages per one shard.

expand

public PCollection<KinesisRecord> expand(PBegin input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PBegin,PCollection<KinesisRecord>>