Package org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryIO.Write<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<T>,WriteResult>

org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write<T>

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

BigQueryIO

public abstract static class BigQueryIO.Write<T>
extends PTransform<PCollection<T>,WriteResult>

Implementation of BigQueryIO.write().

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	BigQueryIO.Write.CreateDisposition	An enumeration type for the BigQuery create disposition strings.
static enum	BigQueryIO.Write.Method	Determines the method used to insert data in BigQuery.
static enum	BigQueryIO.Write.SchemaUpdateOption	An enumeration type for the BigQuery schema update options strings.
static enum	BigQueryIO.Write.WriteDisposition	An enumeration type for the BigQuery write disposition strings.

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor	Description
Write()

Method Summary

Modifier and Type	Method	Description
WriteResult	expand(PCollection<T> input)	Override this method to specify how this PTransform should be expanded on the given InputT.
abstract BigQueryIO.Write.Method	getMethod()
@Nullable ValueProvider<TableReference>	getTable()	Returns the table reference, or null.
BigQueryIO.Write<T>	ignoreInsertIds()	Setting this option to true disables insertId based data deduplication offered by BigQuery.
BigQueryIO.Write<T>	ignoreUnknownValues()	Accept rows that contain values that do not match the schema.
BigQueryIO.Write<T>	optimizedWrites()	If true, enables new codepaths that are expected to use less resources while writing to BigQuery.
void	populateDisplayData(DisplayData.Builder builder)	Register display data for the given transform or component.
BigQueryIO.Write<T>	skipInvalidRows()	Insert all valid rows of a request, even if invalid rows exist.
BigQueryIO.Write<T>	to(TableReference table)	Writes to the given table, specified as a TableReference.
BigQueryIO.Write<T>	to(String tableSpec)	Writes to the given table, specified in the format described in BigQueryHelpers.parseTableSpec(java.lang.String).
BigQueryIO.Write<T>	to(DynamicDestinations<T,?> dynamicDestinations)	Writes to the table and schema specified by the DynamicDestinations object.
BigQueryIO.Write<T>	to(ValueProvider<String> tableSpec)	Same as to(String), but with a ValueProvider.
BigQueryIO.Write<T>	to(SerializableFunction<ValueInSingleWindow<T>,TableDestination> tableFunction)	Writes to table specified by the specified table function.
BigQueryIO.Write<T>	useAvroLogicalTypes()	Enables interpreting logical types into their corresponding types (ie.
BigQueryIO.Write<T>	useBeamSchema()	If true, then the BigQuery schema will be inferred from the input schema.
void	validate(PipelineOptions pipelineOptions)	Called before running the Pipeline to verify this transform is fully and correctly specified.
BigQueryIO.Write<T>	withAutoSchemaUpdate(boolean autoSchemaUpdate)	If true, enables automatically detecting BigQuery table schema updates.
BigQueryIO.Write<T>	withAutoSharding()	If true, enables using a dynamically determined number of shards to write to BigQuery.
BigQueryIO.Write<T>	withAvroFormatFunction(SerializableFunction<AvroWriteRequest<T>,GenericRecord> avroFormatFunction)	Formats the user's type into a GenericRecord to be written to BigQuery.
BigQueryIO.Write<T>	withAvroSchemaFactory(SerializableFunction<@Nullable TableSchema,Schema> avroSchemaFactory)	Uses the specified function to convert a TableSchema to a Schema.
BigQueryIO.Write<T>	withAvroWriter(SerializableFunction<Schema,DatumWriter<T>> writerFactory)	Writes the user's type as avro using the supplied DatumWriter.
<AvroT> BigQueryIO.Write<T>	withAvroWriter(SerializableFunction<AvroWriteRequest<T>,AvroT> avroFormatFunction, SerializableFunction<Schema,DatumWriter<AvroT>> writerFactory)	Convert's the user's type to an avro record using the supplied avroFormatFunction.
BigQueryIO.Write<T>	withBigLakeConfiguration(Map<String,String> bigLakeConfiguration)	Specifies a configuration to create BigLake tables.
BigQueryIO.Write<T>	withClustering()	Allows writing to clustered tables when to(SerializableFunction) or to(DynamicDestinations) is used.
BigQueryIO.Write<T>	withClustering(Clustering clustering)	Specifies the clustering fields to use when writing to a single output table.
BigQueryIO.Write<T>	withCreateDisposition(BigQueryIO.Write.CreateDisposition createDisposition)	Specifies whether the table should be created if it does not exist.
BigQueryIO.Write<T>	withCustomGcsTempLocation(ValueProvider<String> customGcsTempLocation)	Provides a custom location on GCS for storing temporary files to be loaded via BigQuery batch load jobs.
BigQueryIO.Write<T>	withDefaultMissingValueInterpretation(com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation missingValueInterpretation)	Specify how missing values should be interpreted when there is a default value in the schema.
BigQueryIO.Write<T>	withDeterministicRecordIdFn(SerializableFunction<T,String> toUniqueIdFunction)
BigQueryIO.Write<T>	withDirectWriteProtos(boolean directWriteProtos)
BigQueryIO.Write<T>	withErrorHandler(ErrorHandler<BadRecord,?> errorHandler)
BigQueryIO.Write<T>	withExtendedErrorInfo()	Enables extended error information by enabling WriteResult.getFailedInsertsWithErr()
BigQueryIO.Write<T>	withFailedInsertRetryPolicy(InsertRetryPolicy retryPolicy)	Specifies a policy for handling failed inserts.
BigQueryIO.Write<T>	withFormatFunction(SerializableFunction<T,TableRow> formatFunction)	Formats the user's type into a TableRow to be written to BigQuery.
BigQueryIO.Write<T>	withFormatRecordOnFailureFunction(SerializableFunction<T,TableRow> formatFunction)	If an insert failure occurs, this function is applied to the originally supplied T element.
BigQueryIO.Write<T>	withJsonClustering(ValueProvider<String> jsonClustering)	The same as withClustering(Clustering), but takes a JSON-serialized Clustering object in a deferred ValueProvider.
BigQueryIO.Write<T>	withJsonSchema(String jsonSchema)	Similar to withSchema(TableSchema) but takes in a JSON-serialized TableSchema.
BigQueryIO.Write<T>	withJsonSchema(ValueProvider<String> jsonSchema)	Same as withJsonSchema(String) but using a deferred ValueProvider.
BigQueryIO.Write<T>	withJsonTimePartitioning(ValueProvider<String> partitioning)	The same as withTimePartitioning(com.google.api.services.bigquery.model.TimePartitioning), but takes a JSON-serialized object.
BigQueryIO.Write<T>	withKmsKey(String kmsKey)
BigQueryIO.Write<T>	withLoadJobProjectId(String loadJobProjectId)	Set the project the BigQuery load job will be initiated from.
BigQueryIO.Write<T>	withLoadJobProjectId(ValueProvider<String> loadJobProjectId)
BigQueryIO.Write<T>	withMaxBytesPerPartition(long maxBytesPerPartition)	Control how much data will be assigned to a single BigQuery load job.
BigQueryIO.Write<T>	withMaxFileSize(long maxFileSize)	Controls the maximum byte size per file to be loaded into BigQuery.
BigQueryIO.Write<T>	withMaxFilesPerBundle(int maxFilesPerBundle)	Control how many files will be written concurrently by a single worker when using BigQuery load jobs before spilling to a shuffle.
BigQueryIO.Write<T>	withMaxFilesPerPartition(int maxFilesPerPartition)	Controls how many files will be assigned to a single BigQuery load job.
BigQueryIO.Write<T>	withMaxRetryJobs(int maxRetryJobs)	If set, this will set the max number of retry of batch load jobs.
BigQueryIO.Write<T>	withMethod(BigQueryIO.Write.Method method)	Choose the method used to write data to BigQuery.
BigQueryIO.Write<T>	withNumFileShards(int numFileShards)	Control how many file shards are written when using BigQuery load jobs.
BigQueryIO.Write<T>	withNumStorageWriteApiStreams(int numStorageWriteApiStreams)	Control how many parallel streams are used when using Storage API writes.
BigQueryIO.Write<T>	withoutValidation()	Disables BigQuery table validation.
BigQueryIO.Write<T>	withPrimaryKey(List<String> primaryKey)
BigQueryIO.Write<T>	withPropagateSuccessfulStorageApiWrites(boolean propagateSuccessfulStorageApiWrites)	If set to true, then all successful writes will be propagated to WriteResult and accessible via the WriteResult.getSuccessfulStorageApiInserts() method.
BigQueryIO.Write<T>	withPropagateSuccessfulStorageApiWrites(Predicate<String> columnsToPropagate)	If called, then all successful writes will be propagated to WriteResult and accessible via the WriteResult.getSuccessfulStorageApiInserts() method.
BigQueryIO.Write<T>	withRowMutationInformationFn(SerializableFunction<T,RowMutationInformation> updateFn)	Allows upserting and deleting rows for tables with a primary key defined.
BigQueryIO.Write<T>	withSchema(TableSchema schema)	Uses the specified schema for rows to be written.
BigQueryIO.Write<T>	withSchema(ValueProvider<TableSchema> schema)	Same as withSchema(TableSchema) but using a deferred ValueProvider.
BigQueryIO.Write<T>	withSchemaFromView(PCollectionView<Map<String,String>> view)	Allows the schemas for each table to be computed within the pipeline itself.
BigQueryIO.Write<T>	withSchemaUpdateOptions(Set<BigQueryIO.Write.SchemaUpdateOption> schemaUpdateOptions)	Allows the schema of the destination table to be updated as a side effect of the write.
BigQueryIO.Write<T>	withSuccessfulInsertsPropagation(boolean propagateSuccessful)	If true, it enables the propagation of the successfully inserted TableRows on BigQuery as part of the WriteResult object when using BigQueryIO.Write.Method.STREAMING_INSERTS.
BigQueryIO.Write<T>	withTableDescription(String tableDescription)	Specifies the table description.
BigQueryIO.Write<T>	withTestServices(BigQueryServices testServices)
BigQueryIO.Write<T>	withTimePartitioning(TimePartitioning partitioning)	Allows newly created tables to include a TimePartitioning class.
BigQueryIO.Write<T>	withTimePartitioning(ValueProvider<TimePartitioning> partitioning)	Like withTimePartitioning(TimePartitioning) but using a deferred ValueProvider.
BigQueryIO.Write<T>	withTriggeringFrequency(Duration triggeringFrequency)	Choose the frequency at which file writes are triggered.
BigQueryIO.Write<T>	withWriteDisposition(BigQueryIO.Write.WriteDisposition writeDisposition)	Specifies what to do with existing data in the table, in case the table already exists.
BigQueryIO.Write<T>	withWriteTempDataset(String writeTempDataset)	Temporary dataset.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

Write

public Write()

Method Details

getMethod

public abstract BigQueryIO.Write.Method getMethod()

to

public BigQueryIO.Write<T> to(String tableSpec)

Writes to the given table, specified in the format described in BigQueryHelpers.parseTableSpec(java.lang.String).

to

public BigQueryIO.Write<T> to(TableReference table)

Writes to the given table, specified as a TableReference.

to

public BigQueryIO.Write<T> to(ValueProvider<String> tableSpec)

Same as to(String), but with a ValueProvider.

to

public BigQueryIO.Write<T> to(SerializableFunction<ValueInSingleWindow<T>,TableDestination> tableFunction)

Writes to table specified by the specified table function. The table is a function of ValueInSingleWindow, so can be determined by the value or by the window.

If the function produces destinations configured with clustering fields, ensure that withClustering() is also set so that the clustering configurations get properly encoded and decoded.

to

public BigQueryIO.Write<T> to(DynamicDestinations<T,?> dynamicDestinations)

Writes to the table and schema specified by the DynamicDestinations object.

If any of the returned destinations are configured with clustering fields, ensure that the passed DynamicDestinations object returns TableDestinationCoderV3 when DynamicDestinations.getDestinationCoder() is called.

withFormatFunction

public BigQueryIO.Write<T> withFormatFunction(SerializableFunction<T,TableRow> formatFunction)

Formats the user's type into a TableRow to be written to BigQuery.

withFormatRecordOnFailureFunction

public BigQueryIO.Write<T> withFormatRecordOnFailureFunction(SerializableFunction<T,TableRow> formatFunction)

If an insert failure occurs, this function is applied to the originally supplied T element.

For BigQueryIO.Write.Method.STREAMING_INSERTS method, the resulting TableRow will be accessed via WriteResult.getFailedInsertsWithErr().

For BigQueryIO.Write.Method.STORAGE_WRITE_API and BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE methods, the resulting TableRow will be accessed via WriteResult.getFailedStorageApiInserts().

withAvroFormatFunction

public BigQueryIO.Write<T> withAvroFormatFunction(SerializableFunction<AvroWriteRequest<T>,GenericRecord> avroFormatFunction)

Formats the user's type into a GenericRecord to be written to BigQuery. The GenericRecords are written as avro using the standard GenericDatumWriter.

This is mutually exclusive with withFormatFunction(org.apache.beam.sdk.transforms.SerializableFunction<T, com.google.api.services.bigquery.model.TableRow>), only one may be set.

withAvroWriter

public BigQueryIO.Write<T> withAvroWriter(SerializableFunction<Schema,DatumWriter<T>> writerFactory)

Writes the user's type as avro using the supplied DatumWriter.

This is mutually exclusive with withFormatFunction(org.apache.beam.sdk.transforms.SerializableFunction<T, com.google.api.services.bigquery.model.TableRow>), only one may be set.

Overwrites withAvroFormatFunction(org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.io.gcp.bigquery.AvroWriteRequest<T>, org.apache.avro.generic.GenericRecord>) if it has been set.

withAvroWriter

public <AvroT> BigQueryIO.Write<T> withAvroWriter(SerializableFunction<AvroWriteRequest<T>,AvroT> avroFormatFunction,
 SerializableFunction<Schema,DatumWriter<AvroT>> writerFactory)

Convert's the user's type to an avro record using the supplied avroFormatFunction. Records are then written using the supplied writer instances returned from writerFactory.

This is mutually exclusive with withFormatFunction(org.apache.beam.sdk.transforms.SerializableFunction<T, com.google.api.services.bigquery.model.TableRow>), only one may be set.

Overwrites withAvroFormatFunction(org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.io.gcp.bigquery.AvroWriteRequest<T>, org.apache.avro.generic.GenericRecord>) if it has been set.

withAvroSchemaFactory

public BigQueryIO.Write<T> withAvroSchemaFactory(SerializableFunction<@Nullable TableSchema,Schema> avroSchemaFactory)

Uses the specified function to convert a TableSchema to a Schema.

If not specified, the TableSchema will automatically be converted to an avro schema.

withSchema

public BigQueryIO.Write<T> withSchema(TableSchema schema)

Uses the specified schema for rows to be written.

The schema is required only if writing to a table that does not already exist, and BigQueryIO.Write.CreateDisposition is set to BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED.

withSchema

public BigQueryIO.Write<T> withSchema(ValueProvider<TableSchema> schema)

Same as withSchema(TableSchema) but using a deferred ValueProvider.

withJsonSchema

public BigQueryIO.Write<T> withJsonSchema(String jsonSchema)

Similar to withSchema(TableSchema) but takes in a JSON-serialized TableSchema.

withJsonSchema

public BigQueryIO.Write<T> withJsonSchema(ValueProvider<String> jsonSchema)

Same as withJsonSchema(String) but using a deferred ValueProvider.

withSchemaFromView

public BigQueryIO.Write<T> withSchemaFromView(PCollectionView<Map<String,String>> view)

Allows the schemas for each table to be computed within the pipeline itself.

The input is a map-valued PCollectionView mapping string tablespecs to JSON-formatted TableSchemas. Tablespecs must be in the same format as taken by to(String).

withTimePartitioning

public BigQueryIO.Write<T> withTimePartitioning(TimePartitioning partitioning)

Allows newly created tables to include a TimePartitioning class. Can only be used when writing to a single table. If to(SerializableFunction) or to(DynamicDestinations) is used to write dynamic tables, time partitioning can be directly set in the returned TableDestination.

withTimePartitioning

public BigQueryIO.Write<T> withTimePartitioning(ValueProvider<TimePartitioning> partitioning)

Like withTimePartitioning(TimePartitioning) but using a deferred ValueProvider.

withJsonTimePartitioning

public BigQueryIO.Write<T> withJsonTimePartitioning(ValueProvider<String> partitioning)

The same as withTimePartitioning(com.google.api.services.bigquery.model.TimePartitioning), but takes a JSON-serialized object.

withClustering

public BigQueryIO.Write<T> withClustering(Clustering clustering)

Specifies the clustering fields to use when writing to a single output table. If to(SerializableFunction) or to(DynamicDestinations) is used to write to dynamic tables, the fields here will be ignored; call withClustering() instead.

withJsonClustering

public BigQueryIO.Write<T> withJsonClustering(ValueProvider<String> jsonClustering)

The same as withClustering(Clustering), but takes a JSON-serialized Clustering object in a deferred ValueProvider. For example: `"{"fields": ["column1", "column2", "column3"]}"`

withClustering

public BigQueryIO.Write<T> withClustering()

Allows writing to clustered tables when to(SerializableFunction) or to(DynamicDestinations) is used. The returned TableDestination objects should specify the clustering fields per table. If writing to a single table, use withClustering(Clustering) instead to pass a Clustering instance that specifies the static clustering fields to use.

Setting this option enables use of TableDestinationCoderV3 which encodes clustering information. The updated coder is compatible with non-clustered tables, so can be freely set for newly deployed pipelines, but note that pipelines using an older coder must be drained before setting this option, since TableDestinationCoderV3 will not be able to read state written with a previous version.

withCreateDisposition

public BigQueryIO.Write<T> withCreateDisposition(BigQueryIO.Write.CreateDisposition createDisposition)

Specifies whether the table should be created if it does not exist.

withWriteDisposition

public BigQueryIO.Write<T> withWriteDisposition(BigQueryIO.Write.WriteDisposition writeDisposition)

Specifies what to do with existing data in the table, in case the table already exists.

withSchemaUpdateOptions

public BigQueryIO.Write<T> withSchemaUpdateOptions(Set<BigQueryIO.Write.SchemaUpdateOption> schemaUpdateOptions)

Allows the schema of the destination table to be updated as a side effect of the write.

This configuration applies only when writing to BigQuery with BigQueryIO.Write.Method.FILE_LOADS as method.

withTableDescription

public BigQueryIO.Write<T> withTableDescription(String tableDescription)

Specifies the table description.

withBigLakeConfiguration

public BigQueryIO.Write<T> withBigLakeConfiguration(Map<String,String> bigLakeConfiguration)

Specifies a configuration to create BigLake tables. The following options are available:

• connectionId (REQUIRED): the name of your cloud resource connection.
• storageUri (REQUIRED): the path to your GCS folder where data will be written to. This sink will create sub-folders for each project, dataset, and table destination. Example: if you specify a storageUri of "gs://foo/bar" and writing to table "my_project.my_dataset.my_table", your data will be written under "gs://foo/bar/my_project/my_dataset/my_table/"
• fileFormat (OPTIONAL): defaults to "parquet"
• tableFormat (OPTIONAL): defaults to "iceberg"

NOTE: This is only supported with the Storage Write API methods.

See Also:

• BigQuery Tables for Apache Iceberg documentation

withFailedInsertRetryPolicy

public BigQueryIO.Write<T> withFailedInsertRetryPolicy(InsertRetryPolicy retryPolicy)

Specifies a policy for handling failed inserts.

Currently this only is allowed when writing an unbounded collection to BigQuery. Bounded collections are written using batch load jobs, so we don't get per-element failures. Unbounded collections are written using streaming inserts, so we have access to per-element insert results.

withoutValidation

public BigQueryIO.Write<T> withoutValidation()

Disables BigQuery table validation.

withMethod

public BigQueryIO.Write<T> withMethod(BigQueryIO.Write.Method method)

Choose the method used to write data to BigQuery. See the Javadoc on BigQueryIO.Write.Method for information and restrictions of the different methods.

withRowMutationInformationFn

public BigQueryIO.Write<T> withRowMutationInformationFn(SerializableFunction<T,RowMutationInformation> updateFn)

Allows upserting and deleting rows for tables with a primary key defined. Provides a mapping function that determines how a row is applied to BigQuery (upsert, or delete) along with a sequence number for ordering operations.

This is supported when using the BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE insert method, and with either BigQueryIO.Write.CreateDisposition.CREATE_NEVER or BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED. For CREATE_IF_NEEDED, a primary key must be specified using withPrimaryKey(java.util.List<java.lang.String>).

withDirectWriteProtos

public BigQueryIO.Write<T> withDirectWriteProtos(boolean directWriteProtos)

withLoadJobProjectId

public BigQueryIO.Write<T> withLoadJobProjectId(String loadJobProjectId)

Set the project the BigQuery load job will be initiated from. This is only applicable when the write method is set to BigQueryIO.Write.Method.FILE_LOADS. If omitted, the project of the destination table is used.

withLoadJobProjectId

public BigQueryIO.Write<T> withLoadJobProjectId(ValueProvider<String> loadJobProjectId)

withTriggeringFrequency

public BigQueryIO.Write<T> withTriggeringFrequency(Duration triggeringFrequency)

Choose the frequency at which file writes are triggered.

This is only applicable when the write method is set to BigQueryIO.Write.Method.FILE_LOADS or BigQueryIO.Write.Method.STORAGE_WRITE_API, and only when writing an unbounded PCollection.

Every triggeringFrequency duration, a BigQuery load job will be generated for all the data written since the last load job. BigQuery has limits on how many load jobs can be triggered per day, so be careful not to set this duration too low, or you may exceed daily quota. Often this is set to 5 or 10 minutes to ensure that the project stays well under the BigQuery quota. See Quota Policy for more information about BigQuery quotas.

withNumFileShards

public BigQueryIO.Write<T> withNumFileShards(int numFileShards)

Control how many file shards are written when using BigQuery load jobs. Applicable only when also setting withTriggeringFrequency(org.joda.time.Duration). To let runner determine the sharding at runtime, set withAutoSharding() instead.

withNumStorageWriteApiStreams

public BigQueryIO.Write<T> withNumStorageWriteApiStreams(int numStorageWriteApiStreams)

Control how many parallel streams are used when using Storage API writes.

For streaming pipelines, and when withTriggeringFrequency(org.joda.time.Duration) is also set. To let runner determine the sharding at runtime, set this to zero, or withAutoSharding() instead.

For batch pipelines, it inserts a redistribute. To not reshufle and keep the pipeline parallelism as is, set this to zero.

withPropagateSuccessfulStorageApiWrites

public BigQueryIO.Write<T> withPropagateSuccessfulStorageApiWrites(boolean propagateSuccessfulStorageApiWrites)

If set to true, then all successful writes will be propagated to WriteResult and accessible via the WriteResult.getSuccessfulStorageApiInserts() method.

withPropagateSuccessfulStorageApiWrites

public BigQueryIO.Write<T> withPropagateSuccessfulStorageApiWrites(Predicate<String> columnsToPropagate)

If called, then all successful writes will be propagated to WriteResult and accessible via the WriteResult.getSuccessfulStorageApiInserts() method. The predicate allows filtering out columns from appearing in the resulting PCollection. The argument to the predicate is the name of the field to potentially be included in the output. Nested fields will be presented using . notation - e.g. a.b.c. If you want a nested field included, you must ensure that the predicate returns true for every parent field. e.g. if you want field "a.b.c" included, the predicate must return true for "a" for "a.b" and for "a.b.c".

The predicate will be invoked repeatedly for every field in every message, so it is recommended that it be as lightweight as possible. e.g. looking up fields in a hash table instead of searching a list of field names.

withCustomGcsTempLocation

public BigQueryIO.Write<T> withCustomGcsTempLocation(ValueProvider<String> customGcsTempLocation)

Provides a custom location on GCS for storing temporary files to be loaded via BigQuery batch load jobs. See "Usage with templates" in BigQueryIO documentation for discussion.

withExtendedErrorInfo

public BigQueryIO.Write<T> withExtendedErrorInfo()

Enables extended error information by enabling WriteResult.getFailedInsertsWithErr()

ATM this only works if using BigQueryIO.Write.Method.STREAMING_INSERTS. See withMethod(Method).

skipInvalidRows

public BigQueryIO.Write<T> skipInvalidRows()

Insert all valid rows of a request, even if invalid rows exist. This is only applicable when the write method is set to BigQueryIO.Write.Method.STREAMING_INSERTS. The default value is false, which causes the entire request to fail if any invalid rows exist.

ignoreUnknownValues

public BigQueryIO.Write<T> ignoreUnknownValues()

Accept rows that contain values that do not match the schema. The unknown values are ignored. Default is false, which treats unknown values as errors.

useAvroLogicalTypes

public BigQueryIO.Write<T> useAvroLogicalTypes()

Enables interpreting logical types into their corresponding types (ie. TIMESTAMP), instead of only using their raw types (ie. LONG).

ignoreInsertIds

public BigQueryIO.Write<T> ignoreInsertIds()

Setting this option to true disables insertId based data deduplication offered by BigQuery. For more information, please see https://cloud.google.com/bigquery/streaming-data-into-bigquery#disabling_best_effort_de-duplication.

withKmsKey

public BigQueryIO.Write<T> withKmsKey(String kmsKey)

withPrimaryKey

public BigQueryIO.Write<T> withPrimaryKey(List<String> primaryKey)

withDefaultMissingValueInterpretation

public BigQueryIO.Write<T> withDefaultMissingValueInterpretation(com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation missingValueInterpretation)

Specify how missing values should be interpreted when there is a default value in the schema. Options are to take the default value or to write an explicit null (not an option of the field is also required.). Note: this is only used when using one of the storage write API insert methods.

optimizedWrites

public BigQueryIO.Write<T> optimizedWrites()

If true, enables new codepaths that are expected to use less resources while writing to BigQuery. Not enabled by default in order to maintain backwards compatibility.

useBeamSchema

public BigQueryIO.Write<T> useBeamSchema()

If true, then the BigQuery schema will be inferred from the input schema. If no formatFunction is set, then BigQueryIO will automatically turn the input records into TableRows that match the schema.

withAutoSharding

public BigQueryIO.Write<T> withAutoSharding()

If true, enables using a dynamically determined number of shards to write to BigQuery. This can be used for BigQueryIO.Write.Method.FILE_LOADS, BigQueryIO.Write.Method.STREAMING_INSERTS and BigQueryIO.Write.Method.STORAGE_WRITE_API. Only applicable to unbounded data. If using BigQueryIO.Write.Method.FILE_LOADS, numFileShards set via withNumFileShards(int) will be ignored.

withMaxRetryJobs

public BigQueryIO.Write<T> withMaxRetryJobs(int maxRetryJobs)

If set, this will set the max number of retry of batch load jobs.

withSuccessfulInsertsPropagation

public BigQueryIO.Write<T> withSuccessfulInsertsPropagation(boolean propagateSuccessful)

If true, it enables the propagation of the successfully inserted TableRows on BigQuery as part of the WriteResult object when using BigQueryIO.Write.Method.STREAMING_INSERTS. By default this property is set on true. In the cases where a pipeline won't make use of the insert results this property can be set on false, which will make the pipeline let go of those inserted TableRows and reclaim worker resources.

withAutoSchemaUpdate

public BigQueryIO.Write<T> withAutoSchemaUpdate(boolean autoSchemaUpdate)

If true, enables automatically detecting BigQuery table schema updates. Table schema updates are usually noticed within several minutes. Only supported when using one of the STORAGE_API insert methods.

withDeterministicRecordIdFn

public BigQueryIO.Write<T> withDeterministicRecordIdFn(SerializableFunction<T,String> toUniqueIdFunction)

withTestServices

public BigQueryIO.Write<T> withTestServices(BigQueryServices testServices)

withMaxFilesPerBundle

public BigQueryIO.Write<T> withMaxFilesPerBundle(int maxFilesPerBundle)

Control how many files will be written concurrently by a single worker when using BigQuery load jobs before spilling to a shuffle. When data comes into this transform, it is written to one file per destination per worker. When there are more files than maxFilesPerBundle (DEFAULT: 20), the data is shuffled (i.e. Grouped By Destination), and written to files one-by-one-per-worker. This flag sets the maximum number of files that a single worker can write concurrently before shuffling the data. This flag should be used with caution. Setting a high number can increase the memory pressure on workers, and setting a low number can make a pipeline slower (due to the need to shuffle data).

withMaxFileSize

public BigQueryIO.Write<T> withMaxFileSize(long maxFileSize)

Controls the maximum byte size per file to be loaded into BigQuery. If the amount of data written to one file reaches this threshold, we will close that file and continue writing in a new file.

The default value (4 TiB) respects BigQuery's maximum number of source URIs per job configuration.

See Also:

• BigQuery Load Job Limits

withMaxFilesPerPartition

public BigQueryIO.Write<T> withMaxFilesPerPartition(int maxFilesPerPartition)

Controls how many files will be assigned to a single BigQuery load job. If the number of files increases past this threshold, we will spill it over into multiple load jobs as necessary.

The default value (10,000 files) respects BigQuery's maximum number of source URIs per job configuration.

See Also:

• BigQuery Load Job Limits

withMaxBytesPerPartition

public BigQueryIO.Write<T> withMaxBytesPerPartition(long maxBytesPerPartition)

Control how much data will be assigned to a single BigQuery load job. If the amount of data flowing into one BatchLoads partition exceeds this value, that partition will be handled via multiple load jobs.

The default value (11 TiB) respects BigQuery's maximum size per load job limit and is appropriate for most use cases. Reducing the value of this parameter can improve stability when loading to tables with complex schemas containing thousands of fields.

See Also:

• BigQuery Load Job Limits

withWriteTempDataset

public BigQueryIO.Write<T> withWriteTempDataset(String writeTempDataset)

Temporary dataset. When writing to BigQuery from large file loads, the BigQueryIO.write() will create temporary tables in a dataset to store staging data from partitions. With this option, you can set an existing dataset to create the temporary tables. BigQueryIO will create temporary tables in that dataset, and will remove it once it is not needed. No other tables in the dataset will be modified. Remember that the dataset must exist and your job needs permissions to create and remove tables inside that dataset.

withErrorHandler

public BigQueryIO.Write<T> withErrorHandler(ErrorHandler<BadRecord,?> errorHandler)

validate

public void validate(PipelineOptions pipelineOptions)

Description copied from class: PTransform

Called before running the Pipeline to verify this transform is fully and correctly specified.

By default, does nothing.

Overrides:

validate in class PTransform<PCollection<T>,WriteResult>

expand

public WriteResult expand(PCollection<T> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<T>,WriteResult>

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: PTransform

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class PTransform<PCollection<T>,WriteResult>

Parameters:

builder - The builder to populate with display data.

See Also:

• HasDisplayData

getTable

public @Nullable ValueProvider<TableReference> getTable()

Returns the table reference, or null.