org.apache.beam.sdk.transforms.splittabledofn

Class Backlog

java.lang.Object

org.apache.beam.sdk.transforms.splittabledofn.Backlog

public abstract class Backlog
extends java.lang.Object

A representation for the amount of known work represented as a backlog. Note that backlog byte[] representations must be lexicographically comparable. Backlog byte[] representations should preferably represent a linear space and be comparable within the same partition.

It is up to each restriction tracker to convert between their natural representation of outstanding work and this representation. For example:

• Block based file source (e.g. Avro): From the end of the current block, the remaining number of bytes to the end of the restriction represented as a big endian 64 bit unsigned integer.
• Pull based queue based source (e.g. Pubsub): The local/global backlog available in number of messages or number of messages / bytes that have not been processed represented as a big endian 64 bit unsigned integer.
• Key range based source (e.g. Shuffle, Bigtable, ...): Scale the start key to be one and end key to be zero and interpolate the position of the next splittable key as the backlog. If information about the probability density function or cumulative distribution function is available, backlog interpolation can be improved. Alternatively, if the number of encoded bytes for the keys and values is known for the key range, the backlog of remaining bytes can be used.

RestrictionTrackers should implement Backlogs.HasBacklog to report a backlog where the element and restriction pair uniquely identify the resource. Otherwise RestrictionTrackers should implement Backlogs.HasPartitionedBacklog to report a backlog for a shared resource such as a message queue.

See Bundles w/ SplittableDoFns: Backlog & Splitting for further details.

Constructor Summary

Constructors

Constructor and Description
Backlog()

Method Summary

Modifier and Type	Method and Description
java.math.BigDecimal	backlog() Returns the byte[] representation of the backlog if it is known.
protected abstract java.math.BigDecimal	backlogInternal()
boolean	isUnknown() Returns whether this backlog is known or unknown.
static Backlog	of(java.math.BigDecimal backlog) Returns a backlog represented by the specified bytes.
static Backlog	unknown() Returns an unknown backlog.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Backlog

public Backlog()

Method Detail

of

public static Backlog of(java.math.BigDecimal backlog)

Returns a backlog represented by the specified bytes.

unknown

public static Backlog unknown()

Returns an unknown backlog.

isUnknown

public boolean isUnknown()

Returns whether this backlog is known or unknown.

backlog

public java.math.BigDecimal backlog()

Returns the byte[] representation of the backlog if it is known.

Throws:

java.lang.IllegalStateException - if the backlog is unknown.

backlogInternal

@Nullable
protected abstract java.math.BigDecimal backlogInternal()