org.apache.beam.sdk.transforms

Annotation Type DoFn.GetSize

@Documented
 @Retention(value=RUNTIME)
 @Target(value=METHOD)
 @Experimental(value=SPLITTABLE_DO_FN)
public static @interface DoFn.GetSize

Annotation for the method that returns the corresponding size for an element and restriction pair.

Signature: double getSize(<arguments>);

This method must satisfy the following constraints:

• If one of its arguments is tagged with the DoFn.Element annotation, then it will be passed the current element being processed; the argument must be of type InputT. Note that automatic conversion of Rows and DoFn.FieldAccess parameters are currently unsupported.
• If one of its arguments is tagged with the DoFn.Restriction annotation, then it will be passed the current restriction being processed; the argument must be of type RestrictionT.
• If one of its arguments is tagged with the DoFn.Timestamp annotation, then it will be passed the timestamp of the current element being processed; the argument must be of type Instant.
• If one of its arguments is a RestrictionTracker, then it will be passed a tracker that is initialized for the current DoFn.Restriction. The argument must be of the exact type RestrictionTracker<RestrictionT, PositionT>.
• If one of its arguments is a subtype of BoundedWindow, then it will be passed the window of the current element. When applied by ParDo the subtype of BoundedWindow must match the type of windows on the input PCollection. If the window is not accessed a runner may perform additional optimizations.
• If one of its arguments is of type PaneInfo, then it will be passed information about the current triggering pane.
• If one of the parameters is of type PipelineOptions, then it will be passed the options for the current pipeline.

Returns a double representing the size of the current element and restriction.

Splittable DoFns should only provide this method if the default implementation within the RestrictionTracker is an inaccurate representation of known work.

It is up to each splittable to convert between their natural representation of outstanding work and this representation. For example:

• Block based file source (e.g. Avro): From the end of the current block, the remaining number of bytes to the end of the restriction.
• Pull based queue based source (e.g. Pubsub): The local/global size available in number of messages or number of message bytes that have not been processed.
• Key range based source (e.g. Shuffle, Bigtable, ...): Scale the start key to be one and end key to be zero and interpolate the position of the next splittable key as the size. If information about the probability density function or cumulative distribution function is available, size interpolation can be improved. Alternatively, if the number of encoded bytes for the keys and values is known for the key range, the number of remaining bytes can be used.