org.apache.beam.sdk.transforms

Interface DoFn.BundleFinalizer

Enclosing class:

DoFn<InputT,OutputT>

@Experimental(value=SPLITTABLE_DO_FN)
public static interface DoFn.BundleFinalizer

A parameter that is accessible during @StartBundle, @ProcessElement and @FinishBundle that allows the caller to register a callback that will be invoked after the bundle has been successfully completed and the runner has commit the output.

A common usage would be to perform any acknowledgements required by an external system such as acking messages from a message queue since this callback is only invoked after the output of the bundle has been durably persisted by the runner.

Note that a runner may make the output of the bundle available immediately to downstream consumers without waiting for finalization to succeed. For pipelines that are sensitive to duplicate messages, they must perform output deduplication in the pipeline.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	DoFn.BundleFinalizer.Callback An instance of a function that will be invoked after bundle finalization.

Method Summary

Modifier and Type	Method and Description
void	afterBundleCommit(Instant callbackExpiry, DoFn.BundleFinalizer.Callback callback) The provided function will be called after the runner successfully commits the output of a successful bundle.

Method Detail

afterBundleCommit

void afterBundleCommit(Instant callbackExpiry,
                       DoFn.BundleFinalizer.Callback callback)

The provided function will be called after the runner successfully commits the output of a successful bundle. Throwing during finalization represents that bundle finalization may have failed and the runner may choose to attempt finalization again. The provided callbackExpiry controls how long the finalization is valid for before it is garbage collected and no longer able to be invoked.

Note that finalization is best effort and it is expected that the external system will self recover state if finalization never happens or consistently fails. For example, a queue based system that requires message acknowledgement would replay messages if that acknowledgement was never received within the provided time bound.

See Apache Beam Portability API: How to Finalize Bundles for further details.

Parameters:

callbackExpiry - When the finalization callback expires. If the runner cannot commit results and execute the callback within this duration, the callback will not be invoked.

callback - The finalization callback method for the runner to invoke after processing results have been successfully committed.