org.apache.beam.sdk.transforms.splittabledofn

Interface Sizes.HasSize

All Known Implementing Classes:

ByteKeyRangeTracker, OffsetRangeTracker

Enclosing class:

Sizes

public static interface Sizes.HasSize

RestrictionTrackers which can provide a size should implement this interface. Implementations that do not implement this interface will be assumed to have an equivalent size.

Method Summary

Modifier and Type	Method and Description
double	getSize() A representation for the amount of known work represented as a size.

Method Detail

getSize

double getSize()

A representation for the amount of known work represented as a size. Size double representations should preferably represent a linear space and be comparable within the same partition.

It is up to each restriction tracker to convert between their natural representation of outstanding work and this representation. For example:

• Block based file source (e.g. Avro): From the end of the current block, the remaining number of bytes to the end of the restriction.
• Pull based queue based source (e.g. Pubsub): The local/global size available in number of messages or number of message bytes that have not been processed.
• Key range based source (e.g. Shuffle, Bigtable, ...): Scale the start key to be one and end key to be zero and interpolate the position of the next splittable key as the size. If information about the probability density function or cumulative distribution function is available, size interpolation can be improved. Alternatively, if the number of encoded bytes for the keys and values is known for the key range, the number of remaining bytes can be used.

DoFns should provide a method annotated with DoFn.GetPartition to report a partition identifier if the element and restriction represent the size for a shared resource such as a message queue topic. See DoFn.GetPartition for additional details.