BigQueryIO.Write (Apache Beam 2.16.0)

java.lang.Object
- org.apache.beam.sdk.transforms.PTransform<PCollection<T>,WriteResult>
- - org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

BigQueryIO
```
public abstract static class BigQueryIO.Write<T>
extends PTransform<PCollection<T>,WriteResult>
```
Implementation of BigQueryIO.write().

See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`BigQueryIO.Write.CreateDisposition` An enumeration type for the BigQuery create disposition strings.
`static class`	`BigQueryIO.Write.Method` Determines the method used to insert data in BigQuery.
`static class`	`BigQueryIO.Write.WriteDisposition` An enumeration type for the BigQuery write disposition strings.

Field Summary
- Fields inherited from class org.apache.beam.sdk.transforms.PTransform
  name

Constructor Summary

Constructors
Constructor and Description

Write()

Constructors
Constructor and Description
`Write()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`WriteResult`	`expand(PCollection<T> input)` Override this method to specify how this `PTransform` should be expanded on the given `InputT`.
`ValueProvider<TableReference>`	`getTable()` Returns the table reference, or `null`.
`BigQueryIO.Write<T>`	`ignoreUnknownValues()` Accept rows that contain values that do not match the schema.
`BigQueryIO.Write<T>`	`optimizedWrites()` If true, enables new codepaths that are expected to use less resources while writing to BigQuery.
`void`	`populateDisplayData(DisplayData.Builder builder)` Register display data for the given transform or component.
`BigQueryIO.Write<T>`	`skipInvalidRows()` Insert all valid rows of a request, even if invalid rows exist.
`BigQueryIO.Write<T>`	`to(DynamicDestinations<T,?> dynamicDestinations)` Writes to the table and schema specified by the `DynamicDestinations` object.
`BigQueryIO.Write<T>`	`to(SerializableFunction<ValueInSingleWindow<T>,TableDestination> tableFunction)` Writes to table specified by the specified table function.
`BigQueryIO.Write<T>`	`to(java.lang.String tableSpec)` Writes to the given table, specified in the format described in `BigQueryHelpers.parseTableSpec(java.lang.String)`.
`BigQueryIO.Write<T>`	`to(TableReference table)` Writes to the given table, specified as a `TableReference`.
`BigQueryIO.Write<T>`	`to(ValueProvider<java.lang.String> tableSpec)` Same as `to(String)`, but with a `ValueProvider`.
`BigQueryIO.Write<T>`	`useBeamSchema()` If true, then the BigQuery schema will be inferred from the input schema.
`void`	`validate(PipelineOptions pipelineOptions)` Called before running the Pipeline to verify this transform is fully and correctly specified.
`BigQueryIO.Write<T>`	`withClustering()` Allows writing to clustered tables when `to(SerializableFunction)` or `to(DynamicDestinations)` is used.
`BigQueryIO.Write<T>`	`withClustering(Clustering clustering)` Specifies the clustering fields to use when writing to a single output table.
`BigQueryIO.Write<T>`	`withCreateDisposition(BigQueryIO.Write.CreateDisposition createDisposition)` Specifies whether the table should be created if it does not exist.
`BigQueryIO.Write<T>`	`withCustomGcsTempLocation(ValueProvider<java.lang.String> customGcsTempLocation)` Provides a custom location on GCS for storing temporary files to be loaded via BigQuery batch load jobs.
`BigQueryIO.Write<T>`	`withExtendedErrorInfo()` Enables extended error information by enabling `WriteResult.getFailedInsertsWithErr()`
`BigQueryIO.Write<T>`	`withFailedInsertRetryPolicy(InsertRetryPolicy retryPolicy)` Specfies a policy for handling failed inserts.
`BigQueryIO.Write<T>`	`withFormatFunction(SerializableFunction<T,TableRow> formatFunction)` Formats the user's type into a `TableRow` to be written to BigQuery.
`BigQueryIO.Write<T>`	`withJsonSchema(java.lang.String jsonSchema)` Similar to `withSchema(TableSchema)` but takes in a JSON-serialized `TableSchema`.
`BigQueryIO.Write<T>`	`withJsonSchema(ValueProvider<java.lang.String> jsonSchema)` Same as `withJsonSchema(String)` but using a deferred `ValueProvider`.
`BigQueryIO.Write<T>`	`withJsonTimePartitioning(ValueProvider<java.lang.String> partitioning)` The same as `withTimePartitioning(com.google.api.services.bigquery.model.TimePartitioning)`, but takes a JSON-serialized object.
`BigQueryIO.Write<T>`	`withKmsKey(java.lang.String kmsKey)`
`BigQueryIO.Write<T>`	`withLoadJobProjectId(java.lang.String loadJobProjectId)` Set the project the BigQuery load job will be initiated from.
`BigQueryIO.Write<T>`	`withLoadJobProjectId(ValueProvider<java.lang.String> loadJobProjectId)`
`BigQueryIO.Write<T>`	`withMaxFilesPerBundle(int maxFilesPerBundle)` Control how many files will be written concurrently by a single worker when using BigQuery load jobs before spilling to a shuffle.
`BigQueryIO.Write<T>`	`withMethod(BigQueryIO.Write.Method method)` Choose the method used to write data to BigQuery.
`BigQueryIO.Write<T>`	`withNumFileShards(int numFileShards)` Control how many file shards are written when using BigQuery load jobs.
`BigQueryIO.Write<T>`	`withoutValidation()` Disables BigQuery table validation.
`BigQueryIO.Write<T>`	`withSchema(TableSchema schema)` Uses the specified schema for rows to be written.
`BigQueryIO.Write<T>`	`withSchema(ValueProvider<TableSchema> schema)` Same as `withSchema(TableSchema)` but using a deferred `ValueProvider`.
`BigQueryIO.Write<T>`	`withSchemaFromView(PCollectionView<java.util.Map<java.lang.String,java.lang.String>> view)` Allows the schemas for each table to be computed within the pipeline itself.
`BigQueryIO.Write<T>`	`withTableDescription(java.lang.String tableDescription)` Specifies the table description.
`BigQueryIO.Write<T>`	`withTestServices(BigQueryServices testServices)`
`BigQueryIO.Write<T>`	`withTimePartitioning(TimePartitioning partitioning)` Allows newly created tables to include a `TimePartitioning` class.
`BigQueryIO.Write<T>`	`withTimePartitioning(ValueProvider<TimePartitioning> partitioning)` Like `withTimePartitioning(TimePartitioning)` but using a deferred `ValueProvider`.
`BigQueryIO.Write<T>`	`withTriggeringFrequency(Duration triggeringFrequency)` Choose the frequency at which file writes are triggered.
`BigQueryIO.Write<T>`	`withWriteDisposition(BigQueryIO.Write.WriteDisposition writeDisposition)` Specifies what to do with existing data in the table, in case the table already exists.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, toString

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - Write
```
public Write()
```
- Method Detail
  - to
```
public BigQueryIO.Write<T> to(java.lang.String tableSpec)
```
    Writes to the given table, specified in the format described in BigQueryHelpers.parseTableSpec(java.lang.String).
  - to
```
public BigQueryIO.Write<T> to(TableReference table)
```
    Writes to the given table, specified as a TableReference.
  - to
```
public BigQueryIO.Write<T> to(ValueProvider<java.lang.String> tableSpec)
```
    Same as to(String), but with a ValueProvider.
  - to
```
public BigQueryIO.Write<T> to(SerializableFunction<ValueInSingleWindow<T>,TableDestination> tableFunction)
```
    Writes to table specified by the specified table function. The table is a function of ValueInSingleWindow, so can be determined by the value or by the window.
    If the function produces destinations configured with clustering fields, ensure that withClustering() is also set so that the clustering configurations get properly encoded and decoded.
  - to
```
public BigQueryIO.Write<T> to(DynamicDestinations<T,?> dynamicDestinations)
```
    Writes to the table and schema specified by the DynamicDestinations object.
    If any of the returned destinations are configured with clustering fields, ensure that the passed DynamicDestinations object returns TableDestinationCoderV3 when DynamicDestinations.getDestinationCoder() is called.
  - withFormatFunction
```
public BigQueryIO.Write<T> withFormatFunction(SerializableFunction<T,TableRow> formatFunction)
```
    Formats the user's type into a TableRow to be written to BigQuery.
  - withSchema
```
public BigQueryIO.Write<T> withSchema(TableSchema schema)
```
    Uses the specified schema for rows to be written.
    The schema is required only if writing to a table that does not already exist, and BigQueryIO.Write.CreateDisposition is set to BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED.
  - withSchema
```
public BigQueryIO.Write<T> withSchema(ValueProvider<TableSchema> schema)
```
    Same as withSchema(TableSchema) but using a deferred ValueProvider.
  - withJsonSchema
```
public BigQueryIO.Write<T> withJsonSchema(java.lang.String jsonSchema)
```
    Similar to withSchema(TableSchema) but takes in a JSON-serialized TableSchema.
  - withJsonSchema
```
public BigQueryIO.Write<T> withJsonSchema(ValueProvider<java.lang.String> jsonSchema)
```
    Same as withJsonSchema(String) but using a deferred ValueProvider.
  - withSchemaFromView
```
public BigQueryIO.Write<T> withSchemaFromView(PCollectionView<java.util.Map<java.lang.String,java.lang.String>> view)
```
    Allows the schemas for each table to be computed within the pipeline itself.
    The input is a map-valued PCollectionView mapping string tablespecs to JSON-formatted TableSchemas. Tablespecs must be in the same format as taken by to(String).
  - withTimePartitioning
```
public BigQueryIO.Write<T> withTimePartitioning(TimePartitioning partitioning)
```
    Allows newly created tables to include a TimePartitioning class. Can only be used when writing to a single table. If to(SerializableFunction) or to(DynamicDestinations) is used to write dynamic tables, time partitioning can be directly set in the returned TableDestination.
  - withTimePartitioning
```
public BigQueryIO.Write<T> withTimePartitioning(ValueProvider<TimePartitioning> partitioning)
```
    Like withTimePartitioning(TimePartitioning) but using a deferred ValueProvider.
  - withJsonTimePartitioning
```
public BigQueryIO.Write<T> withJsonTimePartitioning(ValueProvider<java.lang.String> partitioning)
```
    The same as withTimePartitioning(com.google.api.services.bigquery.model.TimePartitioning), but takes a JSON-serialized object.
  - withClustering
```
public BigQueryIO.Write<T> withClustering(Clustering clustering)
```
    Specifies the clustering fields to use when writing to a single output table. Can only be used when is set. If to(SerializableFunction) or to(DynamicDestinations) is used to write to dynamic tables, the fields here will be ignored; call withClustering() instead.
  - withClustering
```
public BigQueryIO.Write<T> withClustering()
```
    Allows writing to clustered tables when to(SerializableFunction) or to(DynamicDestinations) is used. The returned TableDestination objects should specify the time partitioning and clustering fields per table. If writing to a single table, use withClustering(Clustering) instead to pass a Clustering instance that specifies the static clustering fields to use.
    Setting this option enables use of TableDestinationCoderV3 which encodes clustering information. The updated coder is compatible with non-clustered tables, so can be freely set for newly deployed pipelines, but note that pipelines using an older coder must be drained before setting this option, since TableDestinationCoderV3 will not be able to read state written with a previous version.
  - withCreateDisposition
```
public BigQueryIO.Write<T> withCreateDisposition(BigQueryIO.Write.CreateDisposition createDisposition)
```
    Specifies whether the table should be created if it does not exist.
  - withWriteDisposition
```
public BigQueryIO.Write<T> withWriteDisposition(BigQueryIO.Write.WriteDisposition writeDisposition)
```
    Specifies what to do with existing data in the table, in case the table already exists.
  - withTableDescription
```
public BigQueryIO.Write<T> withTableDescription(java.lang.String tableDescription)
```
    Specifies the table description.
  - withFailedInsertRetryPolicy
```
public BigQueryIO.Write<T> withFailedInsertRetryPolicy(InsertRetryPolicy retryPolicy)
```
    Specfies a policy for handling failed inserts.
    Currently this only is allowed when writing an unbounded collection to BigQuery. Bounded collections are written using batch load jobs, so we don't get per-element failures. Unbounded collections are written using streaming inserts, so we have access to per-element insert results.
  - withoutValidation
```
public BigQueryIO.Write<T> withoutValidation()
```
    Disables BigQuery table validation.
  - withMethod
```
public BigQueryIO.Write<T> withMethod(BigQueryIO.Write.Method method)
```
    Choose the method used to write data to BigQuery. See the Javadoc on BigQueryIO.Write.Method for information and restrictions of the different methods.
  - withLoadJobProjectId
```
public BigQueryIO.Write<T> withLoadJobProjectId(java.lang.String loadJobProjectId)
```
    Set the project the BigQuery load job will be initiated from. This is only applicable when the write method is set to BigQueryIO.Write.Method.FILE_LOADS. If omitted, the project of the destination table is used.
  - withLoadJobProjectId
```
public BigQueryIO.Write<T> withLoadJobProjectId(ValueProvider<java.lang.String> loadJobProjectId)
```
  - withTriggeringFrequency
```
public BigQueryIO.Write<T> withTriggeringFrequency(Duration triggeringFrequency)
```
    Choose the frequency at which file writes are triggered.
    This is only applicable when the write method is set to BigQueryIO.Write.Method.FILE_LOADS, and only when writing an unbounded PCollection.
    Every triggeringFrequency duration, a BigQuery load job will be generated for all the data written since the last load job. BigQuery has limits on how many load jobs can be triggered per day, so be careful not to set this duration too low, or you may exceed daily quota. Often this is set to 5 or 10 minutes to ensure that the project stays well under the BigQuery quota. See Quota Policy for more information about BigQuery quotas.
  - withNumFileShards
```
@Experimental
public BigQueryIO.Write<T> withNumFileShards(int numFileShards)
```
    Control how many file shards are written when using BigQuery load jobs. Applicable only when also setting withTriggeringFrequency(org.joda.time.Duration).
  - withCustomGcsTempLocation
```
public BigQueryIO.Write<T> withCustomGcsTempLocation(ValueProvider<java.lang.String> customGcsTempLocation)
```
    Provides a custom location on GCS for storing temporary files to be loaded via BigQuery batch load jobs. See "Usage with templates" in BigQueryIO documentation for discussion.
  - withExtendedErrorInfo
```
public BigQueryIO.Write<T> withExtendedErrorInfo()
```
    Enables extended error information by enabling WriteResult.getFailedInsertsWithErr()
    ATM this only works if using BigQueryIO.Write.Method.STREAMING_INSERTS. See withMethod(Method).
  - skipInvalidRows
```
public BigQueryIO.Write<T> skipInvalidRows()
```
    Insert all valid rows of a request, even if invalid rows exist. This is only applicable when the write method is set to BigQueryIO.Write.Method.STREAMING_INSERTS. The default value is false, which causes the entire request to fail if any invalid rows exist.
  - ignoreUnknownValues
```
public BigQueryIO.Write<T> ignoreUnknownValues()
```
    Accept rows that contain values that do not match the schema. The unknown values are ignored. Default is false, which treats unknown values as errors.
  - withKmsKey
```
public BigQueryIO.Write<T> withKmsKey(java.lang.String kmsKey)
```
  - optimizedWrites
```
@Experimental
public BigQueryIO.Write<T> optimizedWrites()
```
    If true, enables new codepaths that are expected to use less resources while writing to BigQuery. Not enabled by default in order to maintain backwards compatibility.
  - useBeamSchema
```
@Experimental
public BigQueryIO.Write<T> useBeamSchema()
```
    If true, then the BigQuery schema will be inferred from the input schema. If no formatFunction is set, then BigQueryIO will automatically turn the input records into TableRows that match the schema.
  - withTestServices
```
public BigQueryIO.Write<T> withTestServices(BigQueryServices testServices)
```
  - withMaxFilesPerBundle
```
public BigQueryIO.Write<T> withMaxFilesPerBundle(int maxFilesPerBundle)
```
    Control how many files will be written concurrently by a single worker when using BigQuery load jobs before spilling to a shuffle. When data comes into this transform, it is written to one file per destination per worker. When there are more files than maxFilesPerBundle (DEFAULT: 20), the data is shuffled (i.e. Grouped By Destination), and written to files one-by-one-per-worker. This flag sets the maximum number of files that a single worker can write concurrently before shuffling the data. This flag should be used with caution. Setting a high number can increase the memory pressure on workers, and setting a low number can make a pipeline slower (due to the need to shuffle data).
  - validate
```
public void validate(PipelineOptions pipelineOptions)
```
    Description copied from class: PTransform
    
    Called before running the Pipeline to verify this transform is fully and correctly specified.
    By default, does nothing.
    
    Overrides:
    
    validate in class PTransform<PCollection<T>,WriteResult>
  - expand
```
public WriteResult expand(PCollection<T> input)
```
    Description copied from class: PTransform
    
    Override this method to specify how this PTransform should be expanded on the given InputT.
    NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.
    Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
    
    Specified by:
    
    expand in class PTransform<PCollection<T>,WriteResult>
  - populateDisplayData
```
public void populateDisplayData(DisplayData.Builder builder)
```
    Description copied from class: PTransform
    
    Register display data for the given transform or component.
    populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.
    By default, does not register any display data. Implementors may override this method to provide their own display data.
    
    Specified by:
    
    populateDisplayData in interface HasDisplayData
    
    Overrides:
    
    populateDisplayData in class PTransform<PCollection<T>,WriteResult>
    
    Parameters:
    
    builder - The builder to populate with display data.
    
    See Also:
    
    HasDisplayData
  - getTable
```
@Nullable
public ValueProvider<TableReference> getTable()
```
    Returns the table reference, or null.

Class BigQueryIO.Write<T>

Nested Class Summary

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

Constructor Summary

Method Summary

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

Methods inherited from class java.lang.Object

Constructor Detail

Write

Method Detail

to

to

to

to

to

withFormatFunction

withSchema

withSchema

withJsonSchema

withJsonSchema

withSchemaFromView

withTimePartitioning

withTimePartitioning

withJsonTimePartitioning

withClustering

withClustering

withCreateDisposition

withWriteDisposition

withTableDescription

withFailedInsertRetryPolicy

withoutValidation

withMethod

withLoadJobProjectId

withLoadJobProjectId

withTriggeringFrequency

withNumFileShards

withCustomGcsTempLocation

withExtendedErrorInfo

skipInvalidRows

ignoreUnknownValues

withKmsKey

optimizedWrites

useBeamSchema

withTestServices

withMaxFilesPerBundle

validate

expand

populateDisplayData

getTable