org.apache.beam.sdk.io.aws2.kinesis

Class KinesisIO.Write<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<T>,KinesisIO.Write.Result>

org.apache.beam.sdk.io.aws2.kinesis.KinesisIO.Write<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

KinesisIO

public abstract static class KinesisIO.Write<T>
extends PTransform<PCollection<T>,KinesisIO.Write.Result>

Implementation of KinesisIO.write().

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KinesisIO.Write.Result Result of KinesisIO.write().

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
KinesisIO.Write.Result	expand(PCollection<T> input) Override this method to specify how this PTransform should be expanded on the given InputT.
KinesisIO.Write<T>	withBatchMaxBytes(int bytes) Max.
KinesisIO.Write<T>	withBatchMaxRecords(int records) Max.
KinesisIO.Write<T>	withClientConfiguration(ClientConfiguration config) Configuration of Kinesis client.
KinesisIO.Write<T>	withConcurrentRequests(int concurrentRequests) Max number of concurrent batch write requests per bundle.
KinesisIO.Write<T>	withPartitioner(KinesisPartitioner<T> partitioner) Specify how to partition records among all stream shards (required).
KinesisIO.Write<T>	withRecordAggregation(java.util.function.Consumer<KinesisIO.RecordAggregation.Builder> aggregation) Enable record aggregation that is compatible with the KPL / KCL.
KinesisIO.Write<T>	withRecordAggregation(KinesisIO.RecordAggregation aggregation) Enable record aggregation that is compatible with the KPL / KCL.
KinesisIO.Write<T>	withRecordAggregationDisabled() Disable KPL / KCL like record aggregation.
KinesisIO.Write<T>	withSerializer(SerializableFunction<T,byte[]> serializer) Specify how to serialize records to bytes on the stream (required).
KinesisIO.Write<T>	withStreamName(java.lang.String streamName) Kinesis stream name which will be used for writing (required).

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

withStreamName

public KinesisIO.Write<T> withStreamName(java.lang.String streamName)

Kinesis stream name which will be used for writing (required).

withBatchMaxRecords

public KinesisIO.Write<T> withBatchMaxRecords(int records)

Max. number of records to send per batch write request.

withBatchMaxBytes

public KinesisIO.Write<T> withBatchMaxBytes(int bytes)

Max. number of bytes to send per batch write request.

Single records that exceed this limit are sent individually. Though, be careful to not violate the AWS API limit of 1MB per request.

This includes both partition keys and data.

withConcurrentRequests

public KinesisIO.Write<T> withConcurrentRequests(int concurrentRequests)

Max number of concurrent batch write requests per bundle.

Note: Concurrency settings above the default have caused a bug in the AWS SDK v2. Therefore, this configuration is currently not exposed to users.

withRecordAggregation

public KinesisIO.Write<T> withRecordAggregation(KinesisIO.RecordAggregation aggregation)

Enable record aggregation that is compatible with the KPL / KCL.

https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation

Note: The aggregation is a lot simpler than the one offered by KPL. It only aggregates records with the same partition key as it's not aware of explicit hash key ranges per shard.

withRecordAggregation

public KinesisIO.Write<T> withRecordAggregation(java.util.function.Consumer<KinesisIO.RecordAggregation.Builder> aggregation)

Enable record aggregation that is compatible with the KPL / KCL.

https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation

Note: The aggregation is a lot simpler than the one offered by KPL. It only aggregates records with the same partition key as it's not aware of explicit hash key ranges per shard.

withRecordAggregationDisabled

public KinesisIO.Write<T> withRecordAggregationDisabled()

Disable KPL / KCL like record aggregation.

withPartitioner

public KinesisIO.Write<T> withPartitioner(KinesisPartitioner<T> partitioner)

Specify how to partition records among all stream shards (required).

The partitioner is critical to distribute new records among all stream shards.

withSerializer

public KinesisIO.Write<T> withSerializer(SerializableFunction<T,byte[]> serializer)

Specify how to serialize records to bytes on the stream (required).

withClientConfiguration

public KinesisIO.Write<T> withClientConfiguration(ClientConfiguration config)

Configuration of Kinesis client.

expand

public KinesisIO.Write.Result expand(PCollection<T> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<T>,KinesisIO.Write.Result>