org.apache.beam.sdk.io.parquet

Class ParquetIO.ReadFiles.BlockTracker

java.lang.Object

org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<OffsetRange,java.lang.Long>

org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker

org.apache.beam.sdk.io.parquet.ParquetIO.ReadFiles.BlockTracker

All Implemented Interfaces:

RestrictionTracker.HasProgress

Enclosing class:

ParquetIO.ReadFiles

public static class ParquetIO.ReadFiles.BlockTracker
extends OffsetRangeTracker

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker

RestrictionTracker.HasProgress, RestrictionTracker.IsBounded, RestrictionTracker.Progress, RestrictionTracker.TruncateResult<RestrictionT>

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker

lastAttemptedOffset, lastClaimedOffset, range

Constructor Summary

Constructors

Constructor and Description
BlockTracker(OffsetRange range, long totalByteSize, long recordCount)

Method Summary

Modifier and Type	Method and Description
RestrictionTracker.Progress	getProgress() A representation for the amount of known completed and known remaining work.
void	makeProgress()

Methods inherited from class org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker

checkDone, currentRestriction, isBounded, toString, tryClaim, trySplit

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

BlockTracker

public BlockTracker(OffsetRange range,
                    long totalByteSize,
                    long recordCount)

Method Detail

makeProgress

public void makeProgress()
                  throws java.io.IOException

Throws:

java.io.IOException

getProgress

public RestrictionTracker.Progress getProgress()

Description copied from interface: RestrictionTracker.HasProgress

A representation for the amount of known completed and known remaining work.

It is up to each restriction tracker to convert between their natural representation of completed and remaining work and the double representation. For example:

• Block based file source (e.g. Avro): The number of bytes from the beginning of the restriction to the current block and the number of bytes from the current block to the end of the restriction.
• Pull based queue based source (e.g. Pubsub): The local/global size available in number of messages or number of message bytes that have processed and the number of messages or number of message bytes that are outstanding.
• Key range based source (e.g. BigQuery, Bigtable, ...): Scale the start key to be one and end key to be zero and interpolate the position of the next splittable key as a position. If information about the probability density function or cumulative distribution function is available, work completed and work remaining interpolation can be improved. Alternatively, if the number of encoded bytes for the keys and values is known for the key range, the number of completed and remaining bytes can be used.

The work completed and work remaining must be of the same scale whether that be number of messages or number of bytes and should never represent two distinct unit types.

Specified by:

getProgress in interface RestrictionTracker.HasProgress

Overrides:

getProgress in class OffsetRangeTracker