org.apache.beam.sdk.transforms.splittabledofn

Class RestrictionTracker<RestrictionT,PositionT>

java.lang.Object

org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<RestrictionT,PositionT>

Direct Known Subclasses:

ByteKeyRangeTracker, OffsetRangeTracker, PeriodicSequence.OutputRangeTracker

@Experimental(value=SPLITTABLE_DO_FN)
public abstract class RestrictionTracker<RestrictionT,PositionT>
extends java.lang.Object

Manages access to the restriction and keeps track of its claimed part for a splittable DoFn.

RestrictionTrackers should implement RestrictionTracker.HasProgress otherwise poor auto-scaling of workers and/or splitting may result if the progress is an inaccurate representation of the known amount of completed and remaining work.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	RestrictionTracker.HasProgress All RestrictionTrackers SHOULD implement this interface to improve auto-scaling and splitting performance.
static class	RestrictionTracker.Progress A representation for the amount of known completed and remaining work.

Constructor Summary

Constructors

Constructor and Description
RestrictionTracker()

Method Summary

Modifier and Type	Method and Description
abstract void	checkDone() Called by the runner after DoFn.ProcessElement returns.
abstract RestrictionT	currentRestriction() Returns a restriction accurately describing the full range of work the current DoFn.ProcessElement call will do, including already completed work.
abstract boolean	tryClaim(PositionT position) Attempts to claim the block of work in the current restriction identified by the given position.
abstract SplitResult<RestrictionT>	trySplit(double fractionOfRemainder) Splits current restriction based on fractionOfRemainder.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RestrictionTracker

public RestrictionTracker()

Method Detail

tryClaim

public abstract boolean tryClaim(PositionT position)

Attempts to claim the block of work in the current restriction identified by the given position. Each claimed position MUST be a valid split point.

If this succeeds, the DoFn MUST execute the entire block of work. If this fails:

• DoFn.ProcessElement MUST return DoFn.ProcessContinuation#stop without performing any additional work or emitting output (note that emitting output or performing work from DoFn.ProcessElement is also not allowed before the first call to this method).
• checkDone() MUST succeed.

currentRestriction

public abstract RestrictionT currentRestriction()

Returns a restriction accurately describing the full range of work the current DoFn.ProcessElement call will do, including already completed work.

trySplit

@Nullable
public abstract SplitResult<RestrictionT> trySplit(double fractionOfRemainder)

Splits current restriction based on fractionOfRemainder.

If splitting the current restriction is possible, the current restriction is split into a primary and residual restriction pair. This invocation updates the currentRestriction() to be the primary restriction effectively having the current DoFn.ProcessElement execution responsible for performing the work that the primary restriction represents. The residual restriction will be executed in a separate DoFn.ProcessElement invocation (likely in a different process). The work performed by executing the primary and residual restrictions as separate DoFn.ProcessElement invocations MUST be equivalent to the work performed as if this split never occurred.

The fractionOfRemainder should be used in a best effort manner to choose a primary and residual restriction based upon the fraction of the remaining work that the current DoFn.ProcessElement invocation is responsible for. For example, if a DoFn.ProcessElement was reading a file with a restriction representing the offset range [100, 200) and has processed up to offset 130 with a fractionOfRemainder of 0.7, the primary and residual restrictions returned would be [100, 179), [179, 200) (note: currentOffset + fractionOfRemainder * remainingWork = 130 + 0.7 * 70 = 179).

fractionOfRemainder = 0 means a checkpoint is required.

The API is recommended to be implemented for a batch pipeline to improve parallel processing performance.

The API is required to be implemented for a streaming pipeline.

Parameters:

fractionOfRemainder - A hint as to the fraction of work the primary restriction should represent based upon the current known remaining amount of work.

Returns:

a SplitResult if a split was possible, otherwise returns null. If the fractionOfRemainder == 0, a null result MUST imply that the restriction tracker is done and there is no more work left to do.

checkDone

public abstract void checkDone()
                        throws java.lang.IllegalStateException

Called by the runner after DoFn.ProcessElement returns.

Must throw an exception with an informative error message, if there is still any unclaimed work remaining in the restriction.

Throws:

java.lang.IllegalStateException