Package org.apache.beam.sdk.io.aws2.kinesis

Class KinesisIO.Write<T>

java.lang.Object
org.apache.beam.sdk.transforms.PTransform<PCollection<T>,KinesisIO.Write.Result>
org.apache.beam.sdk.io.aws2.kinesis.KinesisIO.Write<T>
All Implemented Interfaces:
Serializable, HasDisplayData
Enclosing class:
KinesisIO
public abstract static class KinesisIO.Write<T> extends PTransform<PCollection<T>,KinesisIO.Write.Result>
Implementation of KinesisIO.write().
See Also:

  • Constructor Details

    • Write

      public Write()

  • Method Details

    • withStreamName

      public KinesisIO.Write<T> withStreamName(String streamName)
      Kinesis stream name which will be used for writing (required).

    • withBatchMaxRecords

      public KinesisIO.Write<T> withBatchMaxRecords(int records)
      Max. number of records to send per batch write request.

    • withBatchMaxBytes

      public KinesisIO.Write<T> withBatchMaxBytes(int bytes)
      Max. number of bytes to send per batch write request.

      Single records that exceed this limit are sent individually. Though, be careful to not violate the AWS API limit of 1MB per request.

      This includes both partition keys and data.

    • withConcurrentRequests

      public KinesisIO.Write<T> withConcurrentRequests(int concurrentRequests)
      Max number of concurrent batch write requests per bundle.

      Note: Concurrency settings above the default have caused a bug in the AWS SDK v2. Therefore, this configuration is currently not exposed to users.

    • withRecordAggregation

      public KinesisIO.Write<T> withRecordAggregation(KinesisIO.RecordAggregation aggregation)
      Enable record aggregation that is compatible with the KPL / KCL.

      https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation

    • withRecordAggregation

      public KinesisIO.Write<T> withRecordAggregation(Consumer<KinesisIO.RecordAggregation.Builder> aggregation)
      Enable record aggregation that is compatible with the KPL / KCL.

      https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation

    • withRecordAggregationDisabled

      public KinesisIO.Write<T> withRecordAggregationDisabled()
      Disable KPL / KCL like record aggregation.

    • withPartitioner

      public KinesisIO.Write<T> withPartitioner(KinesisPartitioner<T> partitioner)
      Specify how to partition records among all stream shards (required).

      The partitioner is critical to distribute new records among all stream shards.

    • withSerializer

      public KinesisIO.Write<T> withSerializer(SerializableFunction<T,byte[]> serializer)
      Specify how to serialize records to bytes on the stream (required).

    • withClientConfiguration

      public KinesisIO.Write<T> withClientConfiguration(ClientConfiguration config)
      Configuration of Kinesis client.

    • expand

      public KinesisIO.Write.Result expand(PCollection<T> input)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PCollection<T>,KinesisIO.Write.Result>