org.apache.beam.runners.fnexecution.control

Class SdkHarnessClient.BundleProcessor

java.lang.Object

org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor

Enclosing class:

SdkHarnessClient

public class SdkHarnessClient.BundleProcessor
extends java.lang.Object

A processor capable of creating bundles for some registered BeamFnApi.ProcessBundleDescriptor.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	SdkHarnessClient.BundleProcessor.ActiveBundle An active bundle for a particular BeamFnApi.ProcessBundleDescriptor.

Method Summary

Modifier and Type	Method and Description
SdkHarnessClient.BundleProcessor.ActiveBundle	newBundle(java.util.Map<java.lang.String,RemoteOutputReceiver<?>> outputReceivers, BundleProgressHandler progressHandler) Start a new bundle for the given BeamFnApi.ProcessBundleDescriptor identifier.
SdkHarnessClient.BundleProcessor.ActiveBundle	newBundle(java.util.Map<java.lang.String,RemoteOutputReceiver<?>> outputReceivers, java.util.Map<KV<java.lang.String,java.lang.String>,RemoteOutputReceiver<org.apache.beam.sdk.util.construction.Timer<?>>> timerReceivers, StateRequestHandler stateRequestHandler, BundleProgressHandler progressHandler, BundleFinalizationHandler finalizationHandler, BundleCheckpointHandler checkpointHandler) Start a new bundle for the given BeamFnApi.ProcessBundleDescriptor identifier.
SdkHarnessClient.BundleProcessor.ActiveBundle	newBundle(java.util.Map<java.lang.String,RemoteOutputReceiver<?>> outputReceivers, java.util.Map<KV<java.lang.String,java.lang.String>,RemoteOutputReceiver<org.apache.beam.sdk.util.construction.Timer<?>>> timerReceivers, StateRequestHandler stateRequestHandler, BundleProgressHandler progressHandler, BundleSplitHandler splitHandler, BundleCheckpointHandler checkpointHandler, BundleFinalizationHandler finalizationHandler) Start a new bundle for the given BeamFnApi.ProcessBundleDescriptor identifier.
SdkHarnessClient.BundleProcessor.ActiveBundle	newBundle(java.util.Map<java.lang.String,RemoteOutputReceiver<?>> outputReceivers, StateRequestHandler stateRequestHandler, BundleProgressHandler progressHandler) Start a new bundle for the given BeamFnApi.ProcessBundleDescriptor identifier.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

newBundle

public SdkHarnessClient.BundleProcessor.ActiveBundle newBundle(java.util.Map<java.lang.String,RemoteOutputReceiver<?>> outputReceivers,
                                                               BundleProgressHandler progressHandler)

Start a new bundle for the given BeamFnApi.ProcessBundleDescriptor identifier.

The input channels for the returned SdkHarnessClient.BundleProcessor.ActiveBundle are derived from the instructions in the BeamFnApi.ProcessBundleDescriptor.

NOTE: It is important to SdkHarnessClient.close() each bundle after all elements are emitted.

 try (ActiveBundle bundle = SdkHarnessClient.newBundle(...)) {
   FnDataReceiver<InputT> inputReceiver =
       (FnDataReceiver) bundle.getInputReceivers().get(mainPCollectionId);
   // send all main input elements ...
 }
 

An exception during SdkHarnessClient.close() will be thrown if the bundle requests finalization or attempts to checkpoint by returning a BeamFnApi.DelayedBundleApplication.

newBundle

public SdkHarnessClient.BundleProcessor.ActiveBundle newBundle(java.util.Map<java.lang.String,RemoteOutputReceiver<?>> outputReceivers,
                                                               StateRequestHandler stateRequestHandler,
                                                               BundleProgressHandler progressHandler)

Start a new bundle for the given BeamFnApi.ProcessBundleDescriptor identifier.

The input channels for the returned SdkHarnessClient.BundleProcessor.ActiveBundle are derived from the instructions in the BeamFnApi.ProcessBundleDescriptor.

NOTE: It is important to SdkHarnessClient.close() each bundle after all elements are emitted.

 try (ActiveBundle bundle = SdkHarnessClient.newBundle(...)) {
   FnDataReceiver<InputT> inputReceiver =
       (FnDataReceiver) bundle.getInputReceivers().get(mainPCollectionId);
   // send all main input elements ...
 }
 

An exception during SdkHarnessClient.close() will be thrown if the bundle requests finalization or attempts to checkpoint by returning a BeamFnApi.DelayedBundleApplication.

newBundle

public SdkHarnessClient.BundleProcessor.ActiveBundle newBundle(java.util.Map<java.lang.String,RemoteOutputReceiver<?>> outputReceivers,
                                                               java.util.Map<KV<java.lang.String,java.lang.String>,RemoteOutputReceiver<org.apache.beam.sdk.util.construction.Timer<?>>> timerReceivers,
                                                               StateRequestHandler stateRequestHandler,
                                                               BundleProgressHandler progressHandler,
                                                               BundleFinalizationHandler finalizationHandler,
                                                               BundleCheckpointHandler checkpointHandler)

Start a new bundle for the given BeamFnApi.ProcessBundleDescriptor identifier.

The input channels for the returned SdkHarnessClient.BundleProcessor.ActiveBundle are derived from the instructions in the BeamFnApi.ProcessBundleDescriptor.

NOTE: It is important to SdkHarnessClient.close() each bundle after all elements are emitted.

 try (ActiveBundle bundle = SdkHarnessClient.newBundle(...)) {
   FnDataReceiver<InputT> inputReceiver =
       (FnDataReceiver) bundle.getInputReceivers().get(mainPCollectionId);
   // send all main input elements ...
 }
 

An exception during SdkHarnessClient.close() will be thrown if the bundle requests finalization if BundleFinalizationHandler is null or attempts to checkpoint by returning a BeamFnApi.DelayedBundleApplication .

newBundle

public SdkHarnessClient.BundleProcessor.ActiveBundle newBundle(java.util.Map<java.lang.String,RemoteOutputReceiver<?>> outputReceivers,
                                                               java.util.Map<KV<java.lang.String,java.lang.String>,RemoteOutputReceiver<org.apache.beam.sdk.util.construction.Timer<?>>> timerReceivers,
                                                               StateRequestHandler stateRequestHandler,
                                                               BundleProgressHandler progressHandler,
                                                               BundleSplitHandler splitHandler,
                                                               BundleCheckpointHandler checkpointHandler,
                                                               BundleFinalizationHandler finalizationHandler)

Start a new bundle for the given BeamFnApi.ProcessBundleDescriptor identifier.

The input channels for the returned SdkHarnessClient.BundleProcessor.ActiveBundle are derived from the instructions in the BeamFnApi.ProcessBundleDescriptor.

NOTE: It is important to SdkHarnessClient.close() each bundle after all elements are emitted.

 try (ActiveBundle bundle = SdkHarnessClient.newBundle(...)) {
   FnDataReceiver<InputT> inputReceiver =
       (FnDataReceiver) bundle.getInputReceivers().get(mainPCollectionId);
   // send all elements ...
 }