Package org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryIO.TypedRead<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<T>>

org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead<T>

All Implemented Interfaces:

Serializable, ProjectionProducer<PTransform<PBegin,PCollection<T>>>, HasDisplayData

Enclosing class:

BigQueryIO

public abstract static class BigQueryIO.TypedRead<T>
extends PTransform<PBegin,PCollection<T>>
implements ProjectionProducer<PTransform<PBegin,PCollection<T>>>

Implementation of BigQueryIO.read(SerializableFunction).

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	BigQueryIO.TypedRead.Method	Determines the method used to read data from BigQuery.
static enum	BigQueryIO.TypedRead.QueryPriority	An enumeration type for the priority of a query.

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor	Description
TypedRead()

Method Summary

Modifier and Type	Method	Description
PTransform<PBegin,PCollection<T>>	actuateProjectionPushdown(Map<TupleTag<?>,FieldAccessDescriptor> outputFields)	Actuate a projection pushdown.
PCollection<T>	expand(PBegin input)	Override this method to specify how this PTransform should be expanded on the given InputT.
BigQueryIO.TypedRead<T>	from(TableReference table)	See BigQueryIO.Read.from(TableReference).
BigQueryIO.TypedRead<T>	from(String tableSpec)	See BigQueryIO.Read.from(String).
BigQueryIO.TypedRead<T>	from(ValueProvider<String> tableSpec)	See BigQueryIO.Read.from(ValueProvider).
BigQueryIO.TypedRead<T>	fromQuery(String query)	See BigQueryIO.Read.fromQuery(String).
BigQueryIO.TypedRead<T>	fromQuery(ValueProvider<String> query)	See BigQueryIO.Read.fromQuery(ValueProvider).
abstract BigQueryIO.TypedRead.Method	getMethod()
@Nullable TableReference	getTable()	See BigQueryIO.Read.getTable().
@Nullable ValueProvider<TableReference>	getTableProvider()	See BigQueryIO.Read.getTableProvider().
void	populateDisplayData(DisplayData.Builder builder)	Register display data for the given transform or component.
static <T> void	readSource(PipelineOptions options, TupleTag<T> rowTag, DoFn.MultiOutputReceiver outputReceiver, BoundedSource<T> streamSource, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ErrorHandlingParseFn<T> errorHandlingParseFn, BadRecordRouter badRecordRouter)
boolean	supportsProjectionPushdown()	Whether this supports projection pushdown.
BigQueryIO.TypedRead<T>	useAvroLogicalTypes()
BigQueryIO.TypedRead<T>	usingStandardSql()	See BigQueryIO.Read.usingStandardSql().
void	validate(PipelineOptions options)	Called before running the Pipeline to verify this transform is fully and correctly specified.
BigQueryIO.TypedRead<T>	withBeamRowConverters(TypeDescriptor<T> typeDescriptor, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<T> toRowFn, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<T> fromRowFn)	Sets the functions to convert elements to/from Row objects.
BigQueryIO.TypedRead<T>	withCoder(Coder<T> coder)	Sets a Coder for the result of the parse function.
BigQueryIO.TypedRead<T>	withErrorHandler(ErrorHandler<BadRecord,?> badRecordErrorHandler)
BigQueryIO.TypedRead<T>	withFormat(com.google.cloud.bigquery.storage.v1.DataFormat format)	See DataFormat.
BigQueryIO.TypedRead<T>	withKmsKey(String kmsKey)	For query sources, use this Cloud KMS key to encrypt any temporary tables created.
BigQueryIO.TypedRead<T>	withMethod(BigQueryIO.TypedRead.Method method)	See BigQueryIO.TypedRead.Method.
BigQueryIO.TypedRead<T>	withoutResultFlattening()	See BigQueryIO.Read.withoutResultFlattening().
BigQueryIO.TypedRead<T>	withoutValidation()	See BigQueryIO.Read.withoutValidation().
BigQueryIO.TypedRead<T>	withQueryLocation(String location)	BigQuery geographic location where the query job will be executed.
BigQueryIO.TypedRead<T>	withQueryPriority(BigQueryIO.TypedRead.QueryPriority priority)	See BigQueryIO.TypedRead.QueryPriority.
BigQueryIO.TypedRead<T>	withQueryTempDataset(String queryTempDatasetRef)	Temporary dataset reference when using fromQuery(String).
BigQueryIO.TypedRead<T>	withQueryTempProjectAndDataset(String queryTempProjectRef, String queryTempDatasetRef)	See withQueryTempDataset(String).
BigQueryIO.TypedRead<T>	withRowRestriction(String rowRestriction)	See withRowRestriction(ValueProvider).
BigQueryIO.TypedRead<T>	withRowRestriction(ValueProvider<String> rowRestriction)	Read only rows which match the specified filter, which must be a SQL expression compatible with Google standard SQL.
BigQueryIO.TypedRead<T>	withSelectedFields(List<String> selectedFields)	See withSelectedFields(ValueProvider).
BigQueryIO.TypedRead<T>	withSelectedFields(ValueProvider<List<String>> selectedFields)	Read only the specified fields (columns) from a BigQuery table.
BigQueryIO.TypedRead<T>	withTemplateCompatibility()
BigQueryIO.TypedRead<T>	withTestServices(BigQueryServices testServices)

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

TypedRead

public TypedRead()

Method Details

getMethod

public abstract BigQueryIO.TypedRead.Method getMethod()

validate

public void validate(PipelineOptions options)

Description copied from class: PTransform

Called before running the Pipeline to verify this transform is fully and correctly specified.

By default, does nothing.

Overrides:

validate in class PTransform<PBegin,PCollection<T>>

expand

public PCollection<T> expand(PBegin input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PBegin,PCollection<T>>

readSource

public static <T> void readSource(PipelineOptions options,
 TupleTag<T> rowTag,
 DoFn.MultiOutputReceiver outputReceiver,
 BoundedSource<T> streamSource,
 org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ErrorHandlingParseFn<T> errorHandlingParseFn,
 BadRecordRouter badRecordRouter)
                           throws Exception

Throws:

Exception

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: PTransform

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class PTransform<PBegin,PCollection<T>>

Parameters:

builder - The builder to populate with display data.

See Also:

• HasDisplayData

getTableProvider

public @Nullable ValueProvider<TableReference> getTableProvider()

See BigQueryIO.Read.getTableProvider().

getTable

public @Nullable TableReference getTable()

See BigQueryIO.Read.getTable().

withCoder

public BigQueryIO.TypedRead<T> withCoder(Coder<T> coder)

Sets a Coder for the result of the parse function. This may be required if a coder can not be inferred automatically.

withKmsKey

public BigQueryIO.TypedRead<T> withKmsKey(String kmsKey)

For query sources, use this Cloud KMS key to encrypt any temporary tables created.

withBeamRowConverters

public BigQueryIO.TypedRead<T> withBeamRowConverters(TypeDescriptor<T> typeDescriptor,
 org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<T> toRowFn,
 org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<T> fromRowFn)

Sets the functions to convert elements to/from Row objects.

Setting these conversion functions is necessary to enable Schema support.

from

public BigQueryIO.TypedRead<T> from(String tableSpec)

See BigQueryIO.Read.from(String).

from

public BigQueryIO.TypedRead<T> from(ValueProvider<String> tableSpec)

See BigQueryIO.Read.from(ValueProvider).

fromQuery

public BigQueryIO.TypedRead<T> fromQuery(String query)

See BigQueryIO.Read.fromQuery(String).

fromQuery

public BigQueryIO.TypedRead<T> fromQuery(ValueProvider<String> query)

See BigQueryIO.Read.fromQuery(ValueProvider).

from

public BigQueryIO.TypedRead<T> from(TableReference table)

See BigQueryIO.Read.from(TableReference).

withoutValidation

public BigQueryIO.TypedRead<T> withoutValidation()

See BigQueryIO.Read.withoutValidation().

withoutResultFlattening

public BigQueryIO.TypedRead<T> withoutResultFlattening()

See BigQueryIO.Read.withoutResultFlattening().

usingStandardSql

public BigQueryIO.TypedRead<T> usingStandardSql()

See BigQueryIO.Read.usingStandardSql().

withQueryPriority

public BigQueryIO.TypedRead<T> withQueryPriority(BigQueryIO.TypedRead.QueryPriority priority)

See BigQueryIO.TypedRead.QueryPriority.

withQueryLocation

public BigQueryIO.TypedRead<T> withQueryLocation(String location)

BigQuery geographic location where the query job will be executed. If not specified, Beam tries to determine the location by examining the tables referenced by the query. Location must be specified for queries not executed in US or EU, or when you are reading from an authorized view. See BigQuery Jobs: query.

withQueryTempDataset

public BigQueryIO.TypedRead<T> withQueryTempDataset(String queryTempDatasetRef)

Temporary dataset reference when using fromQuery(String). When reading from a query, BigQuery will create a temporary dataset and a temporary table to store the results of the query. With this option, you can set an existing dataset to create the temporary table. BigQueryIO will create a temporary table in that dataset, and will remove it once it is not needed. No other tables in the dataset will be modified. If your job does not have permissions to create a new dataset, and you want to use fromQuery(String) (for instance, to read from a view), you should use this option. Remember that the dataset must exist and your job needs permissions to create and remove tables inside that dataset.

withQueryTempProjectAndDataset

public BigQueryIO.TypedRead<T> withQueryTempProjectAndDataset(String queryTempProjectRef,
 String queryTempDatasetRef)

See withQueryTempDataset(String).

withMethod

public BigQueryIO.TypedRead<T> withMethod(BigQueryIO.TypedRead.Method method)

See BigQueryIO.TypedRead.Method.

withFormat

public BigQueryIO.TypedRead<T> withFormat(com.google.cloud.bigquery.storage.v1.DataFormat format)

See DataFormat.

withSelectedFields

public BigQueryIO.TypedRead<T> withSelectedFields(List<String> selectedFields)

See withSelectedFields(ValueProvider).

withSelectedFields

public BigQueryIO.TypedRead<T> withSelectedFields(ValueProvider<List<String>> selectedFields)

Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned.

Requires BigQueryIO.TypedRead.Method.DIRECT_READ. Not compatible with fromQuery(String).

withRowRestriction

public BigQueryIO.TypedRead<T> withRowRestriction(String rowRestriction)

See withRowRestriction(ValueProvider).

withRowRestriction

public BigQueryIO.TypedRead<T> withRowRestriction(ValueProvider<String> rowRestriction)

Read only rows which match the specified filter, which must be a SQL expression compatible with Google standard SQL. If no value is specified, then all rows are returned.

Requires BigQueryIO.TypedRead.Method.DIRECT_READ. Not compatible with fromQuery(String).

withErrorHandler

public BigQueryIO.TypedRead<T> withErrorHandler(ErrorHandler<BadRecord,?> badRecordErrorHandler)

withTemplateCompatibility

public BigQueryIO.TypedRead<T> withTemplateCompatibility()

withTestServices

public BigQueryIO.TypedRead<T> withTestServices(BigQueryServices testServices)

useAvroLogicalTypes

public BigQueryIO.TypedRead<T> useAvroLogicalTypes()

supportsProjectionPushdown

public boolean supportsProjectionPushdown()

Description copied from interface: ProjectionProducer

Whether this supports projection pushdown.

Specified by:

supportsProjectionPushdown in interface ProjectionProducer<T>

actuateProjectionPushdown

public PTransform<PBegin,PCollection<T>> actuateProjectionPushdown(Map<TupleTag<?>,FieldAccessDescriptor> outputFields)

Description copied from interface: ProjectionProducer

Actuate a projection pushdown.

Specified by:

actuateProjectionPushdown in interface ProjectionProducer<T>

Parameters:

outputFields - Map keyed by the TupleTag for each output on which pushdown is requested. The value is the FieldAccessDescriptor containing the list of fields needed for that output; fields not present in the descriptor should be dropped.

Returns:

T that implements the projection pushdown. The return value is assumed to be a drop-in replacement for this; it must have all the same functionality. For this reason, T is usually the same class as this.