org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryStorageTableSource<T>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.sdk.io.BoundedSource<T>

org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class BigQueryStorageTableSource<T>
extends BoundedSource<T>

A Source representing reading from a table.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.BoundedSource

BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

Source.Reader<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected BigQueryServices	bqServices
protected @Nullable com.google.cloud.bigquery.storage.v1.DataFormat	format
protected Coder<T>	outputCoder
protected SerializableFunction<SchemaAndRecord,T>	parseFn
protected @Nullable ValueProvider<java.lang.String>	rowRestrictionProvider
protected @Nullable ValueProvider<java.util.List<java.lang.String>>	selectedFieldsProvider

Method Summary

Modifier and Type	Method and Description
static <T> BigQueryStorageTableSource<T>	create(ValueProvider<TableReference> tableRefProvider, com.google.cloud.bigquery.storage.v1.DataFormat format, @Nullable ValueProvider<java.util.List<java.lang.String>> selectedFields, @Nullable ValueProvider<java.lang.String> rowRestriction, SerializableFunction<SchemaAndRecord,T> parseFn, Coder<T> outputCoder, BigQueryServices bqServices, boolean projectionPushdownApplied)
static <T> BigQueryStorageTableSource<T>	create(ValueProvider<TableReference> tableRefProvider, @Nullable ValueProvider<java.util.List<java.lang.String>> selectedFields, @Nullable ValueProvider<java.lang.String> rowRestriction, SerializableFunction<SchemaAndRecord,T> parseFn, Coder<T> outputCoder, BigQueryServices bqServices)
BoundedSource.BoundedReader<T>	createReader(PipelineOptions options) Returns a new BoundedSource.BoundedReader that reads from this source.
long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
Coder<T>	getOutputCoder() Returns the Coder to use for the data read from this source.
protected Table	getTargetTable(BigQueryOptions options) Returns the table to read from at split time.
protected java.lang.String	getTargetTableId(BigQueryOptions options)
void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
java.util.List<org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageStreamSource<T>>	split(long desiredBundleSizeBytes, PipelineOptions options) Splits the source into bundles of approximately desiredBundleSizeBytes.

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

format

protected final @Nullable com.google.cloud.bigquery.storage.v1.DataFormat format

selectedFieldsProvider

protected final @Nullable ValueProvider<java.util.List<java.lang.String>> selectedFieldsProvider

rowRestrictionProvider

protected final @Nullable ValueProvider<java.lang.String> rowRestrictionProvider

parseFn

protected final SerializableFunction<SchemaAndRecord,T> parseFn

outputCoder

protected final Coder<T> outputCoder

bqServices

protected final BigQueryServices bqServices

Method Detail

create

public static <T> BigQueryStorageTableSource<T> create(ValueProvider<TableReference> tableRefProvider,
                                                       com.google.cloud.bigquery.storage.v1.DataFormat format,
                                                       @Nullable ValueProvider<java.util.List<java.lang.String>> selectedFields,
                                                       @Nullable ValueProvider<java.lang.String> rowRestriction,
                                                       SerializableFunction<SchemaAndRecord,T> parseFn,
                                                       Coder<T> outputCoder,
                                                       BigQueryServices bqServices,
                                                       boolean projectionPushdownApplied)

create

public static <T> BigQueryStorageTableSource<T> create(ValueProvider<TableReference> tableRefProvider,
                                                       @Nullable ValueProvider<java.util.List<java.lang.String>> selectedFields,
                                                       @Nullable ValueProvider<java.lang.String> rowRestriction,
                                                       SerializableFunction<SchemaAndRecord,T> parseFn,
                                                       Coder<T> outputCoder,
                                                       BigQueryServices bqServices)

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: Source

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class Source<T>

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData

getEstimatedSizeBytes

public long getEstimatedSizeBytes(PipelineOptions options)
                           throws java.lang.Exception

Description copied from class: BoundedSource

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

If there is no way to estimate the size of the source implementations MAY return 0L.

Specified by:

getEstimatedSizeBytes in class BoundedSource<T>

Throws:

java.lang.Exception

getTargetTableId

protected java.lang.String getTargetTableId(BigQueryOptions options)
                                     throws java.lang.Exception

Throws:

java.lang.Exception

getTargetTable

protected Table getTargetTable(BigQueryOptions options)
                        throws java.lang.Exception

Returns the table to read from at split time. This is currently never an anonymous table, but it can be a named table which was created to hold the results of a query.

Throws:

java.lang.Exception

getOutputCoder

public Coder<T> getOutputCoder()

Description copied from class: Source

Returns the Coder to use for the data read from this source.

Overrides:

getOutputCoder in class Source<T>

split

public java.util.List<org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageStreamSource<T>> split(long desiredBundleSizeBytes,
                                                                                                PipelineOptions options)
                                                                                         throws java.lang.Exception

Description copied from class: BoundedSource

Splits the source into bundles of approximately desiredBundleSizeBytes.

Specified by:

split in class BoundedSource<T>

Throws:

java.lang.Exception

createReader

public BoundedSource.BoundedReader<T> createReader(PipelineOptions options)
                                            throws java.io.IOException

Description copied from class: BoundedSource

Returns a new BoundedSource.BoundedReader that reads from this source.

Specified by:

createReader in class BoundedSource<T>

Throws:

java.io.IOException