org.apache.beam.sdk.fn.data

Class BeamFnDataBufferingOutboundObserver<T>

java.lang.Object

org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver<T>

All Implemented Interfaces:

java.lang.AutoCloseable, CloseableFnDataReceiver<org.apache.beam.sdk.util.WindowedValue<T>>, FnDataReceiver<org.apache.beam.sdk.util.WindowedValue<T>>

public class BeamFnDataBufferingOutboundObserver<T>
extends java.lang.Object
implements CloseableFnDataReceiver<org.apache.beam.sdk.util.WindowedValue<T>>

A buffering outbound FnDataReceiver for the Beam Fn Data API.

Encodes individually consumed elements with the provided Coder producing a single BeamFnApi.Elements message when the buffer threshold is surpassed.

The default buffer threshold can be overridden by specifying the experiment beam_fn_api_data_buffer_limit=<bytes>

TODO: Handle outputting large elements (> 2GiBs). Note that this also applies to the input side as well.

TODO: Handle outputting elements that are zero bytes by outputting a single byte as a marker, detect on the input side that no bytes were read and force reading a single byte.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	BEAM_FN_API_DATA_BUFFER_LIMIT

Method Summary

Modifier and Type	Method and Description
void	accept(org.apache.beam.sdk.util.WindowedValue<T> t)
void	close() .
void	flush() Eagerly flushes any data that is buffered in this channel.
static <T> BeamFnDataBufferingOutboundObserver<T>	forLocation(LogicalEndpoint endpoint, Coder<org.apache.beam.sdk.util.WindowedValue<T>> coder, org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver<org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements> outboundObserver)
static <T> BeamFnDataBufferingOutboundObserver<T>	forLocationWithBufferLimit(int bufferLimit, LogicalEndpoint endpoint, Coder<org.apache.beam.sdk.util.WindowedValue<T>> coder, org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver<org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements> outboundObserver)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BEAM_FN_API_DATA_BUFFER_LIMIT

public static final java.lang.String BEAM_FN_API_DATA_BUFFER_LIMIT

See Also:

Constant Field Values

Method Detail

forLocation

public static <T> BeamFnDataBufferingOutboundObserver<T> forLocation(LogicalEndpoint endpoint,
                                                                     Coder<org.apache.beam.sdk.util.WindowedValue<T>> coder,
                                                                     org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver<org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements> outboundObserver)

forLocationWithBufferLimit

public static <T> BeamFnDataBufferingOutboundObserver<T> forLocationWithBufferLimit(int bufferLimit,
                                                                                    LogicalEndpoint endpoint,
                                                                                    Coder<org.apache.beam.sdk.util.WindowedValue<T>> coder,
                                                                                    org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver<org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements> outboundObserver)

close

public void close()
           throws java.lang.Exception

Description copied from interface: CloseableFnDataReceiver

.

Does nothing if this CloseableFnDataReceiver is already closed.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface CloseableFnDataReceiver<org.apache.beam.sdk.util.WindowedValue<T>>

Throws:

java.lang.Exception

flush

public void flush()
           throws java.io.IOException

Description copied from interface: CloseableFnDataReceiver

Eagerly flushes any data that is buffered in this channel.

Specified by:

flush in interface CloseableFnDataReceiver<org.apache.beam.sdk.util.WindowedValue<T>>

Throws:

java.io.IOException

accept

public void accept(org.apache.beam.sdk.util.WindowedValue<T> t)
            throws java.io.IOException

Specified by:

accept in interface FnDataReceiver<org.apache.beam.sdk.util.WindowedValue<T>>

Throws:

java.io.IOException