Package org.apache.beam.sdk.transforms

Class AsyncWrapper<K,InputT,OutputT>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<KV<K,InputT>,OutputT>

org.apache.beam.sdk.transforms.AsyncWrapper<K,InputT,OutputT>

All Implemented Interfaces:

Serializable, HasDisplayData

public class AsyncWrapper<K,InputT,OutputT>
extends DoFn<KV<K,InputT>,OutputT>

Class that wraps a dofn and converts it from one which process elements synchronously to one which processes them asynchronously.

For synchronous dofns the default settings mean that many (100s) of elements will be processed in parallel and that processing an element will block all other work on that key. In addition runners are optimized for latencies less than a few seconds and longer operations can result in high retry rates. Async should be considered when the default parallelism is not correct and/or items are expected to take longer than a few seconds to process.

/* NOTE: 1) The wrapped syncFn REQUIRES thread-safety if BOTH parallelism > 1 and the DoFn is stateful. 2) Tagged output multi-outputs are unsupported. 3) StartBundle/finishBundle are invoked per element so any batching or aggregation logic will not behave as expected.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

DoFn.AlwaysFetched, DoFn.BoundedPerElement, DoFn.BundleFinalizer, DoFn.CurrentRecordId, DoFn.CurrentRecordOffset, DoFn.Element, DoFn.FieldAccess, DoFn.FinishBundle, DoFn.FinishBundleContext, DoFn.FireTimestamp, DoFn.GetInitialRestriction, DoFn.GetInitialWatermarkEstimatorState, DoFn.GetRestrictionCoder, DoFn.GetSize, DoFn.GetWatermarkEstimatorStateCoder, DoFn.Key, DoFn.MultiOutputReceiver, DoFn.NewTracker, DoFn.NewWatermarkEstimator, DoFn.OnTimer, DoFn.OnTimerContext, DoFn.OnTimerFamily, DoFn.OnWindowExpiration, DoFn.OnWindowExpirationContext, DoFn.OutputReceiver<T>, DoFn.ProcessContext, DoFn.ProcessContinuation, DoFn.ProcessElement, DoFn.RequiresStableInput, DoFn.RequiresTimeSortedInput, DoFn.Restriction, DoFn.Setup, DoFn.SideInput, DoFn.SplitRestriction, DoFn.StartBundle, DoFn.StartBundleContext, DoFn.StateId, DoFn.Teardown, DoFn.TimerFamily, DoFn.TimerId, DoFn.Timestamp, DoFn.TruncateRestriction, DoFn.UnboundedPerElement, DoFn.WatermarkEstimatorState, DoFn.WindowedContext

Constructor Summary

Constructors

Constructor	Description
AsyncWrapper(DoFn<InputT,OutputT> syncFn, int parallelism, Duration timerFrequency, @Nullable Integer maxItemsToBuffer, @Nullable Duration timeout, @Nullable Duration maxWaitTime, @Nullable SerializableFunction<InputT,Object> idFn, boolean useThreadPool)
AsyncWrapper(DoFn<InputT,OutputT> syncFn, int parallelism, Duration timerFrequency, @Nullable Integer maxItemsToBuffer, @Nullable Duration timeout, @Nullable Duration maxWaitTime, @Nullable SerializableFunction<InputT,Object> idFn, boolean useThreadPool, @Nullable Coder<KV<K,InputT>> coder)

Method Summary

Modifier and Type	Method	Description
void	onTimer(DoFn<KV<K,InputT>,OutputT>.OnTimerContext c, BagState<KV<K,InputT>> toProcessState, Timer timer, DoFn.OutputReceiver<OutputT> receiver)
void	processElement(DoFn<KV<K,InputT>,OutputT>.ProcessContext c, BoundedWindow window, BagState<KV<K,InputT>> toProcessState, Timer timer)
void	setup(PipelineOptions options)
void	teardown()

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

AsyncWrapper

public AsyncWrapper(DoFn<InputT,OutputT> syncFn,
 int parallelism,
 Duration timerFrequency,
 @Nullable Integer maxItemsToBuffer,
 @Nullable Duration timeout,
 @Nullable Duration maxWaitTime,
 @Nullable SerializableFunction<InputT,Object> idFn,
 boolean useThreadPool)

AsyncWrapper

public AsyncWrapper(DoFn<InputT,OutputT> syncFn,
 int parallelism,
 Duration timerFrequency,
 @Nullable Integer maxItemsToBuffer,
 @Nullable Duration timeout,
 @Nullable Duration maxWaitTime,
 @Nullable SerializableFunction<InputT,Object> idFn,
 boolean useThreadPool,
 @Nullable Coder<KV<K,InputT>> coder)

Method Details

setup

@Setup
public void setup(PipelineOptions options)

teardown

@Teardown
public void teardown()

processElement

@ProcessElement
public void processElement(DoFn<KV<K,InputT>,OutputT>.ProcessContext c,
 BoundedWindow window,
 @StateId("to_process")
 BagState<KV<K,InputT>> toProcessState,
 @TimerId("timer")
 Timer timer)

onTimer

@OnTimer("timer")
public void onTimer(DoFn<KV<K,InputT>,OutputT>.OnTimerContext c,
 @StateId("to_process")
 BagState<KV<K,InputT>> toProcessState,
 @TimerId("timer")
 Timer timer,
 DoFn.OutputReceiver<OutputT> receiver)