Package org.apache.beam.sdk.io.aws2.kinesis

Interface KinesisPartitioner<T>

All Superinterfaces:
Serializable
All Known Subinterfaces:
KinesisPartitioner.ExplicitPartitioner<T>
public interface KinesisPartitioner<T> extends Serializable
Kinesis interface for custom partitioner.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Interface
    Description
    static interface 
    KinesisPartitioner.ExplicitPartitioner<T>
    An explicit partitioner that always returns a Nonnull explicit hash key.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final BigInteger
    MAX_HASH_KEY
     
    static final BigInteger
    MIN_HASH_KEY
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static <T> KinesisPartitioner<T>
    explicitRandomPartitioner(int shards)
    Explicit hash key partitioner that randomly returns one of x precalculated hash keys.
    default String
    getExplicitHashKey(T record)
    Optional hash value (128-bit integer) to determine explicitly the shard a record is assigned to based on the hash key range of each shard.
    String
    getPartitionKey(T record)
    Determines which shard in the stream the record is assigned to.

  • Field Details

    • MIN_HASH_KEY

      static final BigInteger MIN_HASH_KEY

    • MAX_HASH_KEY

      static final BigInteger MAX_HASH_KEY

  • Method Details

    • getPartitionKey

      @Nonnull String getPartitionKey(T record)
      Determines which shard in the stream the record is assigned to. Partition keys are Unicode strings with a maximum length limit of 256 characters for each key. Amazon Kinesis Data Streams uses the partition key as input to a hash function that maps the partition key and associated data to a specific shard.

    • getExplicitHashKey

      @Nullable default String getExplicitHashKey(T record)
      Optional hash value (128-bit integer) to determine explicitly the shard a record is assigned to based on the hash key range of each shard. The explicit hash key overrides the partition key hash.

    • explicitRandomPartitioner

      static <T> KinesisPartitioner<T> explicitRandomPartitioner(int shards)
      Explicit hash key partitioner that randomly returns one of x precalculated hash keys. Hash keys are derived by equally dividing the 128-bit hash universe, assuming that hash ranges of shards are also equally sized.

      Note: This simple approach is likely not applicable anymore after resharding a stream. In that case it is recommended to use the ListShards API to retrieve the actual hash key range of each shard and partition based on that.

      See Also: