Package org.apache.beam.sdk.io.aws2.kinesis

Interface KinesisIOOptions

All Superinterfaces:

HasDisplayData, PipelineOptions

public interface KinesisIOOptions
extends PipelineOptions

PipelineOptions for KinesisIO.

Allows passing modify-able configurations in cases when some runner implementation persists KinesisIO.Read serialized objects. Adding new configurations to this class should be exceptional, and standard KinesisIO.Read / KinesisIO.Write should be preferred in most of the cases.

This class appeared during the implementation of EFO consumer. In Flink runner, KinesisIO.Read is serialized with the entire KinesisSource object which was a trouble for EFO feature design: if consumer ARN is part of KinesisIO.Read object, when started from a Flink savepoint, consumer ARN string or null value would be forced from the savepoint.

Consequences of this are:

1. Once a Kinesis source is started, its consumer ARN can't be changed without loosing state (checkpoint-ed shard progress).
2. Kinesis source can not have seamless enabling / disabling of EFO feature without loosing state (checkpoint-ed shard progress).

This PipelineOptions extension allows having modifiable configurations for UnboundedSource.split(int, PipelineOptions) and UnboundedSource.createReader(PipelineOptions, UnboundedSource.CheckpointMark), which is essential for seamless EFO switch on / off.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	KinesisIOOptions.KinesisIOOptionsRegistrar	A registrar containing the default KinesisIOOptions.
static class	KinesisIOOptions.MapFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Method Summary

Modifier and Type	Method	Description
Map<String,String>	getKinesisIOConsumerArns()	Used to enable / disable EFO.
void	setKinesisIOConsumerArns(Map<String,String> value)

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Method Details

getKinesisIOConsumerArns

@InstanceFactory(MapFactory.class)
Map<String,String> getKinesisIOConsumerArns()

Used to enable / disable EFO.

Example:

--kinesisIOConsumerArns={
   "stream-01": "arn:aws:kinesis:...:stream/stream-01/consumer/consumer-01:1678576714",
   "stream-02": "arn:aws:kinesis:...:stream/stream-02/consumer/my-consumer:1679576982",
   ...
 }

setKinesisIOConsumerArns

void setKinesisIOConsumerArns(Map<String,String> value)