org.apache.beam.sdk.fn.data

Class BeamFnDataOutboundAggregator

java.lang.Object

org.apache.beam.sdk.fn.data.BeamFnDataOutboundAggregator

@NotThreadSafe
public class BeamFnDataOutboundAggregator
extends java.lang.Object

An outbound data buffering aggregator with size-based buffer and time-based buffer if corresponding options are set.

The default size-based buffer threshold can be overridden by specifying the experiment data_buffer_size_limit=<bytes>

The default time-based buffer threshold can be overridden by specifying the experiment data_buffer_time_limit_ms=<milliseconds>

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DATA_BUFFER_SIZE_LIMIT
static java.lang.String	DATA_BUFFER_TIME_LIMIT_MS
static int	DEFAULT_BUFFER_LIMIT_BYTES
static long	DEFAULT_BUFFER_LIMIT_TIME_MS

Constructor Summary

Constructors

Constructor and Description
BeamFnDataOutboundAggregator(PipelineOptions options, java.util.function.Supplier<java.lang.String> processBundleRequestIdSupplier, org.apache.beam.vendor.grpc.v1p43p2.io.grpc.stub.StreamObserver<org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements> outboundObserver, boolean collectElementsIfNoFlushes)

Method Summary

Modifier and Type	Method and Description
void	discard()
<T> FnDataReceiver<T>	registerOutputDataLocation(java.lang.String pTransformId, Coder<T> coder) Register the outbound data logical endpoint, returns the FnDataReceiver for processing the endpoint's outbound data.
<T> FnDataReceiver<T>	registerOutputTimersLocation(java.lang.String pTransformId, java.lang.String timerFamilyId, Coder<T> coder) Register the outbound timers logical endpoint, returns the FnDataReceiver for processing the endpoint's outbound timers data.
void	sendElements(org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements elements)
org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements	sendOrCollectBufferedDataAndFinishOutboundStreams() Closes the streams for all registered outbound endpoints.
void	start() Starts the flushing daemon thread if data_buffer_time_limit_ms is set.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DATA_BUFFER_SIZE_LIMIT

public static final java.lang.String DATA_BUFFER_SIZE_LIMIT

See Also:

Constant Field Values

DEFAULT_BUFFER_LIMIT_BYTES

public static final int DEFAULT_BUFFER_LIMIT_BYTES

See Also:

Constant Field Values

DATA_BUFFER_TIME_LIMIT_MS

public static final java.lang.String DATA_BUFFER_TIME_LIMIT_MS

See Also:

Constant Field Values

DEFAULT_BUFFER_LIMIT_TIME_MS

public static final long DEFAULT_BUFFER_LIMIT_TIME_MS

See Also:

Constant Field Values

Constructor Detail

BeamFnDataOutboundAggregator

public BeamFnDataOutboundAggregator(PipelineOptions options,
                                    java.util.function.Supplier<java.lang.String> processBundleRequestIdSupplier,
                                    org.apache.beam.vendor.grpc.v1p43p2.io.grpc.stub.StreamObserver<org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements> outboundObserver,
                                    boolean collectElementsIfNoFlushes)

Method Detail

start

public void start()

Starts the flushing daemon thread if data_buffer_time_limit_ms is set.

registerOutputDataLocation

public <T> FnDataReceiver<T> registerOutputDataLocation(java.lang.String pTransformId,
                                                        Coder<T> coder)

Register the outbound data logical endpoint, returns the FnDataReceiver for processing the endpoint's outbound data.

registerOutputTimersLocation

public <T> FnDataReceiver<T> registerOutputTimersLocation(java.lang.String pTransformId,
                                                          java.lang.String timerFamilyId,
                                                          Coder<T> coder)

Register the outbound timers logical endpoint, returns the FnDataReceiver for processing the endpoint's outbound timers data.

sendOrCollectBufferedDataAndFinishOutboundStreams

public org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements sendOrCollectBufferedDataAndFinishOutboundStreams()

Closes the streams for all registered outbound endpoints. Should be called at the end of each bundle. Returns the buffered Elements if the BeamFnDataOutboundAggregator started with collectElementsIfNoFlushes=true, and there was no previous flush in this bundle, otherwise returns null.

sendElements

public void sendElements(org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements elements)

discard

public void discard()