org.apache.beam.sdk.fn.stream

Class DataStreams

java.lang.Object

org.apache.beam.sdk.fn.stream.DataStreams

public class DataStreams
extends java.lang.Object

inbound(Iterator) treats multiple ByteStrings as a single input stream and outbound(OutputChunkConsumer) treats a single OutputStream as multiple ByteStrings.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataStreams.BlockingQueueIterator<T> Allows for one or more writing threads to append values to this iterator while one reading thread reads values.
static class	DataStreams.DataStreamDecoder<T> An adapter which converts an InputStream to an Iterator of T values using the specified Coder.
static class	DataStreams.ElementDelimitedOutputStream An adapter which wraps an DataStreams.OutputChunkConsumer as an OutputStream.
static interface	DataStreams.OutputChunkConsumer<T> A callback which is invoked whenever the outbound(org.apache.beam.sdk.fn.stream.DataStreams.OutputChunkConsumer<org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ByteString>) OutputStream becomes full.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_OUTBOUND_BUFFER_LIMIT_BYTES

Constructor Summary

Constructors

Constructor and Description
DataStreams()

Method Summary

Modifier and Type	Method and Description
static java.io.InputStream	inbound(java.util.Iterator<org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ByteString> bytes) Converts multiple ByteStrings into a single InputStream.
static DataStreams.ElementDelimitedOutputStream	outbound(DataStreams.OutputChunkConsumer<org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ByteString> consumer) Converts a single element delimited OutputStream into multiple ByteStrings.
static DataStreams.ElementDelimitedOutputStream	outbound(DataStreams.OutputChunkConsumer<org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ByteString> consumer, int maximumChunkSize) Converts a single element delimited OutputStream into multiple ByteStrings using the specified maximum chunk size.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_OUTBOUND_BUFFER_LIMIT_BYTES

public static final int DEFAULT_OUTBOUND_BUFFER_LIMIT_BYTES

See Also:

Constant Field Values

Constructor Detail

DataStreams

public DataStreams()

Method Detail

inbound

public static java.io.InputStream inbound(java.util.Iterator<org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ByteString> bytes)

Converts multiple ByteStrings into a single InputStream.

The iterator is accessed lazily. The supplied Iterator should block until either it knows that no more values will be provided or it has the next ByteString.

Note that this InputStream follows the Beam Fn API specification for forcing values that decode consuming zero bytes to consuming exactly one byte.

outbound

public static DataStreams.ElementDelimitedOutputStream outbound(DataStreams.OutputChunkConsumer<org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ByteString> consumer)

Converts a single element delimited OutputStream into multiple ByteStrings.

Note that users must call DataStreams.ElementDelimitedOutputStream.delimitElement() after each element.

Note that this OutputStream follows the Beam Fn API specification for forcing values that encode producing zero bytes to produce exactly one byte.

outbound

public static DataStreams.ElementDelimitedOutputStream outbound(DataStreams.OutputChunkConsumer<org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ByteString> consumer,
                                                                int maximumChunkSize)

Converts a single element delimited OutputStream into multiple ByteStrings using the specified maximum chunk size.

Note that users must call DataStreams.ElementDelimitedOutputStream.delimitElement() after each element.

Note that this OutputStream follows the Beam Fn API specification for forcing values that encode producing zero bytes to produce exactly one byte.