org.apache.beam.sdk.extensions.ordered

Class OrderedProcessingHandler<EventT,KeyT,StateT extends MutableState<EventT,?>,ResultT>

java.lang.Object

org.apache.beam.sdk.extensions.ordered.OrderedProcessingHandler<EventT,KeyT,StateT,ResultT>

Type Parameters:

EventT - type of events to be processed

KeyT - type of keys which will be used to group the events

StateT - type of internal State which will be used for processing

ResultT - type of the result of the processing which will be output

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

OrderedProcessingHandler.OrderedProcessingGlobalSequenceHandler

public abstract class OrderedProcessingHandler<EventT,KeyT,StateT extends MutableState<EventT,?>,ResultT>
extends java.lang.Object
implements java.io.Serializable

Parent class for Ordered Processing configuration handlers.

There are two types of processing - when the sequence numbers are contiguous per key and these sequences per keys are independent of each other, and when there is a global sequence shared by all keys. In case of the global sequence processing the custom handler must extend from OrderedProcessingHandler.OrderedProcessingGlobalSequenceHandler.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	OrderedProcessingHandler.OrderedProcessingGlobalSequenceHandler<EventT,KeyT,StateT extends MutableState<EventT,?>,ResultT> Parent class for Ordered Processing configuration handlers to handle processing of the events where global sequence is used.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAX_ELEMENTS_TO_OUTPUT

Constructor Summary

Constructors

Constructor and Description
OrderedProcessingHandler(java.lang.Class<EventT> eventTClass, java.lang.Class<KeyT> keyTClass, java.lang.Class<StateT> stateTClass, java.lang.Class<ResultT> resultTClass) Provide concrete classes which will be used by the ordered processing transform.

Method Summary

Modifier and Type	Method and Description
@NonNull Coder<EventT>	getEventCoder(Pipeline pipeline, Coder<KV<KeyT,KV<java.lang.Long,EventT>>> inputCoder) Provide the event coder.
abstract @NonNull EventExaminer<EventT,StateT>	getEventExaminer()
@NonNull Coder<KeyT>	getKeyCoder(Pipeline pipeline, Coder<KV<KeyT,KV<java.lang.Long,EventT>>> inputCoder) Provide the key coder.
int	getMaxOutputElementsPerBundle() Returns the maximum number of elements which will be output per each bundle.
@NonNull Coder<ResultT>	getResultCoder(Pipeline pipeline) Provide the result coder.
@NonNull Coder<StateT>	getStateCoder(Pipeline pipeline) Provide the state coder.
@Nullable Duration	getStatusUpdateFrequency() Determines the frequency of emission of the OrderedProcessingStatus elements.
boolean	isProduceStatusUpdateOnEveryEvent() Indicates if the status update needs to be sent after each event's processing.
void	setMaxOutputElementsPerBundle(int maxOutputElementsPerBundle) Overrides the default value.
void	setProduceStatusUpdateOnEveryEvent(boolean value) Sets the indicator of whether the status notification needs to be produced on every event.
void	setStatusUpdateFrequency(Duration statusUpdateFrequency) Changes the default status update frequency.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_MAX_ELEMENTS_TO_OUTPUT

public static final int DEFAULT_MAX_ELEMENTS_TO_OUTPUT

See Also:

Constant Field Values

Constructor Detail

OrderedProcessingHandler

public OrderedProcessingHandler(java.lang.Class<EventT> eventTClass,
                                java.lang.Class<KeyT> keyTClass,
                                java.lang.Class<StateT> stateTClass,
                                java.lang.Class<ResultT> resultTClass)

Provide concrete classes which will be used by the ordered processing transform.

Parameters:

eventTClass - class of the events

keyTClass - class of the keys

stateTClass - class of the state

resultTClass - class of the results

Method Detail

getEventExaminer

public abstract @NonNull EventExaminer<EventT,StateT> getEventExaminer()

Returns:

the event examiner instance which will be used by the transform.

getEventCoder

public @NonNull Coder<EventT> getEventCoder(Pipeline pipeline,
                                            Coder<KV<KeyT,KV<java.lang.Long,EventT>>> inputCoder)
                                     throws CannotProvideCoderException

Provide the event coder.

The default implementation of the method will use the event coder from the input PCollection. If the input PCollection doesn't use KVCoder, it will attempt to get the coder from the pipeline's coder registry.

Parameters:

pipeline - of the transform

inputCoder - input coder of the transform

Returns:

event coder

Throws:

CannotProvideCoderException - if the method can't determine the coder based on the above algorithm.

getStateCoder

public @NonNull Coder<StateT> getStateCoder(Pipeline pipeline)
                                     throws CannotProvideCoderException

Provide the state coder.

The default implementation will attempt to get the coder from the pipeline's code registry.

Parameters:

pipeline - of the transform

Returns:

the state coder

Throws:

CannotProvideCoderException

getKeyCoder

public @NonNull Coder<KeyT> getKeyCoder(Pipeline pipeline,
                                        Coder<KV<KeyT,KV<java.lang.Long,EventT>>> inputCoder)
                                 throws CannotProvideCoderException

Provide the key coder.

The default implementation of the method will use the event coder from the input PCollection. If the input PCollection doesn't use KVCoder, it will attempt to get the coder from the pipeline's coder registry.

Parameters:

pipeline -

inputCoder -

Returns:

Throws:

CannotProvideCoderException - if the method can't determine the coder based on the above algorithm.

getResultCoder

public @NonNull Coder<ResultT> getResultCoder(Pipeline pipeline)
                                       throws CannotProvideCoderException

Provide the result coder.

The default implementation will attempt to get the coder from the pipeline's code registry.

Parameters:

pipeline -

Returns:

result coder

Throws:

CannotProvideCoderException

getStatusUpdateFrequency

public @Nullable Duration getStatusUpdateFrequency()

Determines the frequency of emission of the OrderedProcessingStatus elements.

Default is 5 seconds.

Returns:

the frequency of updates. If null is returned, no updates will be emitted on a scheduled basis.

setStatusUpdateFrequency

public void setStatusUpdateFrequency(Duration statusUpdateFrequency)

Changes the default status update frequency. Updates will be disabled if set to null.

Parameters:

statusUpdateFrequency -

isProduceStatusUpdateOnEveryEvent

public boolean isProduceStatusUpdateOnEveryEvent()

Indicates if the status update needs to be sent after each event's processing.

Default is false.

Returns:

See Also:

getStatusUpdateFrequency, PCollection of processing statuses

setProduceStatusUpdateOnEveryEvent

public void setProduceStatusUpdateOnEveryEvent(boolean value)

Sets the indicator of whether the status notification needs to be produced on every event.

Parameters:

value -

getMaxOutputElementsPerBundle

public int getMaxOutputElementsPerBundle()

Returns the maximum number of elements which will be output per each bundle. The default is 10,000 elements.

This is used to limit the amount of data produced for each bundle - many runners have limitations on how much data can be output from a single bundle. If many events arrive out of sequence and are buffered then at some point a single event can cause processing of a large number of buffered events.

Returns:

setMaxOutputElementsPerBundle

public void setMaxOutputElementsPerBundle(int maxOutputElementsPerBundle)

Overrides the default value.

Parameters:

maxOutputElementsPerBundle -