org.apache.beam.sdk.io.kinesis

Class KinesisIO.Write

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<byte[]>,PDone>

org.apache.beam.sdk.io.kinesis.KinesisIO.Write

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

KinesisIO

public abstract static class KinesisIO.Write
extends PTransform<PCollection<byte[]>,PDone>

Implementation of KinesisIO.write().

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<byte[]> input) Override this method to specify how this PTransform should be expanded on the given InputT.
KinesisIO.Write	withAWSClientsProvider(AWSClientsProvider awsClientsProvider) Allows to specify custom AWSClientsProvider.
KinesisIO.Write	withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider, Regions region) Specify AWSCredentialsProvider and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider, Regions region, java.lang.String serviceEndpoint) Specify AWSCredentialsProvider and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider, Regions region, java.lang.String serviceEndpoint, boolean verifyCertificate) Specify credential details and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, Regions region) Specify credential details and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, Regions region, java.lang.String serviceEndpoint) Specify credential details and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, Regions region, java.lang.String serviceEndpoint, boolean verifyCertificate) Specify credential details and region to be used to write to Kinesis.
KinesisIO.Write	withPartitioner(KinesisPartitioner partitioner) Allows to specify custom implementation of KinesisPartitioner.
KinesisIO.Write	withPartitionKey(java.lang.String partitionKey) Specify default partition key.
KinesisIO.Write	withProducerProperties(java.util.Properties properties) Specify the configuration properties for Kinesis Producer Library (KPL).
KinesisIO.Write	withStreamName(java.lang.String streamName) Specify Kinesis stream name which will be used for writing, this name is required.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

withStreamName

public KinesisIO.Write withStreamName(java.lang.String streamName)

Specify Kinesis stream name which will be used for writing, this name is required.

withPartitionKey

public KinesisIO.Write withPartitionKey(java.lang.String partitionKey)

Specify default partition key.

In case if you need to define more complicated logic for key partitioning then you can create your own implementation of KinesisPartitioner and specify it by withPartitioner(KinesisPartitioner)

Using one of the methods withPartitioner(KinesisPartitioner) or withPartitionKey(String) is required but not both in the same time.

withPartitioner

public KinesisIO.Write withPartitioner(KinesisPartitioner partitioner)

Allows to specify custom implementation of KinesisPartitioner.

This method should be used to balance a distribution of new written records among all stream shards.

Using one of the methods withPartitioner(KinesisPartitioner) or withPartitionKey(String) is required but not both in the same time.

withProducerProperties

public KinesisIO.Write withProducerProperties(java.util.Properties properties)

Specify the configuration properties for Kinesis Producer Library (KPL).

Example of creating new KPL configuration:

Properties properties = new Properties(); properties.setProperty("CollectionMaxCount", "1000"); properties.setProperty("ConnectTimeout", "10000");

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(AWSClientsProvider awsClientsProvider)

Allows to specify custom AWSClientsProvider. AWSClientsProvider creates new IKinesisProducer which is later used for writing to Kinesis.

This method should be used if withAWSClientsProvider(AWSCredentialsProvider, Regions) does not suit well.

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider,
                                              Regions region)

Specify AWSCredentialsProvider and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(java.lang.String awsAccessKey,
                                              java.lang.String awsSecretKey,
                                              Regions region)

Specify credential details and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider,
                                              Regions region,
                                              java.lang.String serviceEndpoint)

Specify AWSCredentialsProvider and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(java.lang.String awsAccessKey,
                                              java.lang.String awsSecretKey,
                                              Regions region,
                                              java.lang.String serviceEndpoint)

Specify credential details and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(AWSCredentialsProvider awsCredentialsProvider,
                                              Regions region,
                                              java.lang.String serviceEndpoint,
                                              boolean verifyCertificate)

Specify credential details and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

The verifyCertificate disables or enables certificate verification. Never set it to false in production.

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(java.lang.String awsAccessKey,
                                              java.lang.String awsSecretKey,
                                              Regions region,
                                              java.lang.String serviceEndpoint,
                                              boolean verifyCertificate)

Specify credential details and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

The verifyCertificate disables or enables certificate verification. Never set it to false in production.

expand

public PDone expand(PCollection<byte[]> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<byte[]>,PDone>