org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryStorageStreamSource<T>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.sdk.io.BoundedSource<T>

org.apache.beam.sdk.io.OffsetBasedSource<T>

org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageStreamSource<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

@Experimental(value=SOURCE_SINK)
public class BigQueryStorageStreamSource<T>
extends OffsetBasedSource<T>

A Source representing a single stream in a read session.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigQueryStorageStreamSource.BigQueryStorageStreamReader<T> A Source.Reader which reads records from a stream.

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.OffsetBasedSource

OffsetBasedSource.OffsetBasedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.BoundedSource

BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

Source.Reader<T>

Method Summary

Modifier and Type	Method and Description
static <T> BigQueryStorageStreamSource<T>	create(com.google.cloud.bigquery.storage.v1beta1.Storage.ReadSession readSession, com.google.cloud.bigquery.storage.v1beta1.Storage.Stream stream, TableSchema tableSchema, SerializableFunction<SchemaAndRecord,T> parseFn, Coder<T> outputCoder, BigQueryServices bqServices)
BigQueryStorageStreamSource.BigQueryStorageStreamReader<T>	createReader(PipelineOptions options) Returns a new BoundedSource.BoundedReader that reads from this source.
OffsetBasedSource<T>	createSourceForSubrange(long start, long end) Returns an OffsetBasedSource for a subrange of the current source.
long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
long	getMaxEndOffset(PipelineOptions options) Returns the actual ending offset of the current source.
Coder<T>	getOutputCoder() Returns the Coder to use for the data read from this source.
void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
java.util.List<? extends OffsetBasedSource<T>>	split(long desiredBundleSizeBytes, PipelineOptions options) Splits the source into bundles of approximately desiredBundleSizeBytes.

Methods inherited from class org.apache.beam.sdk.io.OffsetBasedSource

getBytesPerOffset, getEndOffset, getMinBundleSize, getStartOffset, toString, validate

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

create

public static <T> BigQueryStorageStreamSource<T> create(com.google.cloud.bigquery.storage.v1beta1.Storage.ReadSession readSession,
                                                        com.google.cloud.bigquery.storage.v1beta1.Storage.Stream stream,
                                                        TableSchema tableSchema,
                                                        SerializableFunction<SchemaAndRecord,T> parseFn,
                                                        Coder<T> outputCoder,
                                                        BigQueryServices bqServices)

getOutputCoder

public Coder<T> getOutputCoder()

Description copied from class: Source

Returns the Coder to use for the data read from this source.

Overrides:

getOutputCoder in class Source<T>

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: Source

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class OffsetBasedSource<T>

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData

getEstimatedSizeBytes

public long getEstimatedSizeBytes(PipelineOptions options)

Description copied from class: BoundedSource

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

If there is no way to estimate the size of the source implementations MAY return 0L.

Overrides:

getEstimatedSizeBytes in class OffsetBasedSource<T>

split

public java.util.List<? extends OffsetBasedSource<T>> split(long desiredBundleSizeBytes,
                                                            PipelineOptions options)

Description copied from class: BoundedSource

Splits the source into bundles of approximately desiredBundleSizeBytes.

Overrides:

split in class OffsetBasedSource<T>

getMaxEndOffset

public long getMaxEndOffset(PipelineOptions options)

Description copied from class: OffsetBasedSource

Returns the actual ending offset of the current source. The value returned by this function will be used to clip the end of the range [startOffset, endOffset) such that the range used is [startOffset, min(endOffset, maxEndOffset)).

As an example in which OffsetBasedSource is used to implement a file source, suppose that this source was constructed with an endOffset of Long.MAX_VALUE to indicate that a file should be read to the end. Then this function should determine the actual, exact size of the file in bytes and return it.

Specified by:

getMaxEndOffset in class OffsetBasedSource<T>

createSourceForSubrange

public OffsetBasedSource<T> createSourceForSubrange(long start,
                                                    long end)

Description copied from class: OffsetBasedSource

Returns an OffsetBasedSource for a subrange of the current source. The subrange [start, end) must be within the range [startOffset, endOffset) of the current source, i.e. startOffset <= start < end <= endOffset.

Specified by:

createSourceForSubrange in class OffsetBasedSource<T>

createReader

public BigQueryStorageStreamSource.BigQueryStorageStreamReader<T> createReader(PipelineOptions options)
                                                                        throws java.io.IOException

Description copied from class: BoundedSource

Returns a new BoundedSource.BoundedReader that reads from this source.

Specified by:

createReader in class BoundedSource<T>

Throws:

java.io.IOException