@Experimental(value=PORTABILITY) public static interface DoFn.BundleFinalizer
@FinishBundlethat allows the caller to register a callback that will be invoked after the bundle has been successfully completed and the runner has commit the output.
A common usage would be to perform any acknowledgements required by an external system such as acking messages from a message queue since this callback is only invoked after the output of the bundle has been durably persisted by the runner.
Note that a runner may make the output of the bundle available immediately to downstream consumers without waiting for finalization to succeed. For pipelines that are sensitive to duplicate messages, they must perform output deduplication in the pipeline.
|Modifier and Type||Interface and Description|
An instance of a function that will be invoked after bundle finalization.
|Modifier and Type||Method and Description|
The provided function will be called after the runner successfully commits the output of a successful bundle.
void afterBundleCommit(Instant callbackExpiry, DoFn.BundleFinalizer.Callback callback)
callbackExpirycontrols how long the finalization is valid for before it is garbage collected and no longer able to be invoked.
Note that finalization is best effort and it is expected that the external system will self recover state if finalization never happens or consistently fails. For example, a queue based system that requires message acknowledgement would replay messages if that acknowledgement was never received within the provided time bound.
See Apache Beam Portability API: How to Finalize Bundles for further details.
callbackExpiry- When the finalization callback expires. If the runner cannot commit results and execute the callback within this duration, the callback will not be invoked.
callback- The finalization callback method for the runner to invoke after processing results have been successfully committed.