BigQueryIO.Write (Apache Beam 2.14.0)

java.lang.Object
- org.apache.beam.sdk.transforms.PTransform<PCollection<T>,WriteResult>
- - org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write<T>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

BigQueryIO
```
public abstract static class BigQueryIO.Write<T>
extends PTransform<PCollection<T>,WriteResult>
```
Implementation of BigQueryIO.write().

See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`BigQueryIO.Write.CreateDisposition` An enumeration type for the BigQuery create disposition strings.
`static class`	`BigQueryIO.Write.Method` Determines the method used to insert data in BigQuery.
`static class`	`BigQueryIO.Write.WriteDisposition` An enumeration type for the BigQuery write disposition strings.

Field Summary
- Fields inherited from class org.apache.beam.sdk.transforms.PTransform
  name

Constructor Summary

Constructors
Constructor and Description

Write()

Constructors
Constructor and Description
`Write()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`WriteResult`	`expand(PCollection<T> input)` Override this method to specify how this `PTransform` should be expanded on the given `InputT`.
`ValueProvider<TableReference>`	`getTable()` Returns the table reference, or `null`.
`BigQueryIO.Write<T>`	`ignoreUnknownValues()` Accept rows that contain values that do not match the schema.
`BigQueryIO.Write<T>`	`optimizedWrites()` If true, enables new codepaths that are expected to use less resources while writing to BigQuery.
`void`	`populateDisplayData(DisplayData.Builder builder)` Register display data for the given transform or component.
`BigQueryIO.Write<T>`	`skipInvalidRows()` Insert all valid rows of a request, even if invalid rows exist.
`BigQueryIO.Write<T>`	`to(DynamicDestinations<T,?> dynamicDestinations)` Writes to the table and schema specified by the `DynamicDestinations` object.
`BigQueryIO.Write<T>`	`to(SerializableFunction<ValueInSingleWindow<T>,TableDestination> tableFunction)` Writes to table specified by the specified table function.
`BigQueryIO.Write<T>`	`to(java.lang.String tableSpec)` Writes to the given table, specified in the format described in `BigQueryHelpers.parseTableSpec(java.lang.String)`.
`BigQueryIO.Write<T>`	`to(TableReference table)` Writes to the given table, specified as a `TableReference`.
`BigQueryIO.Write<T>`	`to(ValueProvider<java.lang.String> tableSpec)` Same as `to(String)`, but with a `ValueProvider`.
`BigQueryIO.Write<T>`	`useBeamSchema()` If true, then the BigQuery schema will be inferred from the input schema.
`void`	`validate(PipelineOptions pipelineOptions)` Called before running the Pipeline to verify this transform is fully and correctly specified.
`BigQueryIO.Write<T>`	`withCreateDisposition(BigQueryIO.Write.CreateDisposition createDisposition)` Specifies whether the table should be created if it does not exist.
`BigQueryIO.Write<T>`	`withCustomGcsTempLocation(ValueProvider<java.lang.String> customGcsTempLocation)` Provides a custom location on GCS for storing temporary files to be loaded via BigQuery batch load jobs.
`BigQueryIO.Write<T>`	`withExtendedErrorInfo()` Enables extended error information by enabling `WriteResult.getFailedInsertsWithErr()`
`BigQueryIO.Write<T>`	`withFailedInsertRetryPolicy(InsertRetryPolicy retryPolicy)` Specfies a policy for handling failed inserts.
`BigQueryIO.Write<T>`	`withFormatFunction(SerializableFunction<T,TableRow> formatFunction)` Formats the user's type into a `TableRow` to be written to BigQuery.
`BigQueryIO.Write<T>`	`withJsonSchema(java.lang.String jsonSchema)` Similar to `withSchema(TableSchema)` but takes in a JSON-serialized `TableSchema`.
`BigQueryIO.Write<T>`	`withJsonSchema(ValueProvider<java.lang.String> jsonSchema)` Same as `withJsonSchema(String)` but using a deferred `ValueProvider`.
`BigQueryIO.Write<T>`	`withJsonTimePartitioning(ValueProvider<java.lang.String> partitioning)` The same as `withTimePartitioning(com.google.api.services.bigquery.model.TimePartitioning)`, but takes a JSON-serialized object.
`BigQueryIO.Write<T>`	`withKmsKey(java.lang.String kmsKey)`
`BigQueryIO.Write<T>`	`withLoadJobProjectId(java.lang.String loadJobProjectId)` Set the project the BigQuery load job will be initiated from.
`BigQueryIO.Write<T>`	`withLoadJobProjectId(ValueProvider<java.lang.String> loadJobProjectId)`
`BigQueryIO.Write<T>`	`withMethod(BigQueryIO.Write.Method method)` Choose the method used to write data to BigQuery.
`BigQueryIO.Write<T>`	`withNumFileShards(int numFileShards)` Control how many file shards are written when using BigQuery load jobs.
`BigQueryIO.Write<T>`	`withoutValidation()` Disables BigQuery table validation.
`BigQueryIO.Write<T>`	`withSchema(TableSchema schema)` Uses the specified schema for rows to be written.
`BigQueryIO.Write<T>`	`withSchema(ValueProvider<TableSchema> schema)` Same as `withSchema(TableSchema)` but using a deferred `ValueProvider`.
`BigQueryIO.Write<T>`	`withSchemaFromView(PCollectionView<java.util.Map<java.lang.String,java.lang.String>> view)` Allows the schemas for each table to be computed within the pipeline itself.
`BigQueryIO.Write<T>`	`withTableDescription(java.lang.String tableDescription)` Specifies the table description.
`BigQueryIO.Write<T>`	`withTestServices(BigQueryServices testServices)`
`BigQueryIO.Write<T>`	`withTimePartitioning(TimePartitioning partitioning)` Allows newly created tables to include a `TimePartitioning` class.
`BigQueryIO.Write<T>`	`withTimePartitioning(ValueProvider<TimePartitioning> partitioning)` Like `withTimePartitioning(TimePartitioning)` but using a deferred `ValueProvider`.
`BigQueryIO.Write<T>`	`withTriggeringFrequency(Duration triggeringFrequency)` Choose the frequency at which file writes are triggered.
`BigQueryIO.Write<T>`	`withWriteDisposition(BigQueryIO.Write.WriteDisposition writeDisposition)` Specifies what to do with existing data in the table, in case the table already exists.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, toString

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - Write
```
public Write()
```
- Method Detail
  - to
```
public BigQueryIO.Write<T> to(java.lang.String tableSpec)
```
    Writes to the given table, specified in the format described in BigQueryHelpers.parseTableSpec(java.lang.String).
  - to
```
public BigQueryIO.Write<T> to(TableReference table)
```
    Writes to the given table, specified as a TableReference.
  - to
```
public BigQueryIO.Write<T> to(ValueProvider<java.lang.String> tableSpec)
```
    Same as to(String), but with a ValueProvider.
  - to
```
public BigQueryIO.Write<T> to(SerializableFunction<ValueInSingleWindow<T>,TableDestination> tableFunction)
```
    Writes to table specified by the specified table function. The table is a function of ValueInSingleWindow, so can be determined by the value or by the window.
  - to
```
public BigQueryIO.Write<T> to(DynamicDestinations<T,?> dynamicDestinations)
```
    Writes to the table and schema specified by the DynamicDestinations object.
  - withFormatFunction
```
public BigQueryIO.Write<T> withFormatFunction(SerializableFunction<T,TableRow> formatFunction)
```
    Formats the user's type into a TableRow to be written to BigQuery.
  - withSchema
```
public BigQueryIO.Write<T> withSchema(TableSchema schema)
```
    Uses the specified schema for rows to be written.
    The schema is required only if writing to a table that does not already exist, and BigQueryIO.Write.CreateDisposition is set to BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED.
  - withSchema
```
public BigQueryIO.Write<T> withSchema(ValueProvider<TableSchema> schema)
```
    Same as withSchema(TableSchema) but using a deferred ValueProvider.
  - withJsonSchema
```
public BigQueryIO.Write<T> withJsonSchema(java.lang.String jsonSchema)
```
    Similar to withSchema(TableSchema) but takes in a JSON-serialized TableSchema.
  - withJsonSchema
```
public BigQueryIO.Write<T> withJsonSchema(ValueProvider<java.lang.String> jsonSchema)
```
    Same as withJsonSchema(String) but using a deferred ValueProvider.
  - withSchemaFromView
```
public BigQueryIO.Write<T> withSchemaFromView(PCollectionView<java.util.Map<java.lang.String,java.lang.String>> view)
```
    Allows the schemas for each table to be computed within the pipeline itself.
    The input is a map-valued PCollectionView mapping string tablespecs to JSON-formatted TableSchemas. Tablespecs must be in the same format as taken by to(String).
  - withTimePartitioning
```
public BigQueryIO.Write<T> withTimePartitioning(TimePartitioning partitioning)
```
    Allows newly created tables to include a TimePartitioning class. Can only be used when writing to a single table. If to(SerializableFunction) or to(DynamicDestinations) is used to write dynamic tables, time partitioning can be directly in the returned TableDestination.
  - withTimePartitioning
```
public BigQueryIO.Write<T> withTimePartitioning(ValueProvider<TimePartitioning> partitioning)
```
    Like withTimePartitioning(TimePartitioning) but using a deferred ValueProvider.
  - withJsonTimePartitioning
```
public BigQueryIO.Write<T> withJsonTimePartitioning(ValueProvider<java.lang.String> partitioning)
```
    The same as withTimePartitioning(com.google.api.services.bigquery.model.TimePartitioning), but takes a JSON-serialized object.
  - withCreateDisposition
```
public BigQueryIO.Write<T> withCreateDisposition(BigQueryIO.Write.CreateDisposition createDisposition)
```
    Specifies whether the table should be created if it does not exist.
  - withWriteDisposition
```
public BigQueryIO.Write<T> withWriteDisposition(BigQueryIO.Write.WriteDisposition writeDisposition)
```
    Specifies what to do with existing data in the table, in case the table already exists.
  - withTableDescription
```
public BigQueryIO.Write<T> withTableDescription(java.lang.String tableDescription)
```
    Specifies the table description.
  - withFailedInsertRetryPolicy
```
public BigQueryIO.Write<T> withFailedInsertRetryPolicy(InsertRetryPolicy retryPolicy)
```
    Specfies a policy for handling failed inserts.
    Currently this only is allowed when writing an unbounded collection to BigQuery. Bounded collections are written using batch load jobs, so we don't get per-element failures. Unbounded collections are written using streaming inserts, so we have access to per-element insert results.
  - withoutValidation
```
public BigQueryIO.Write<T> withoutValidation()
```
    Disables BigQuery table validation.
  - withMethod
```
public BigQueryIO.Write<T> withMethod(BigQueryIO.Write.Method method)
```
    Choose the method used to write data to BigQuery. See the Javadoc on BigQueryIO.Write.Method for information and restrictions of the different methods.
  - withLoadJobProjectId
```
public BigQueryIO.Write<T> withLoadJobProjectId(java.lang.String loadJobProjectId)
```
    Set the project the BigQuery load job will be initiated from. This is only applicable when the write method is set to BigQueryIO.Write.Method.FILE_LOADS. If omitted, the project of the destination table is used.
  - withLoadJobProjectId
```
public BigQueryIO.Write<T> withLoadJobProjectId(ValueProvider<java.lang.String> loadJobProjectId)
```
  - withTriggeringFrequency
```
public BigQueryIO.Write<T> withTriggeringFrequency(Duration triggeringFrequency)
```
    Choose the frequency at which file writes are triggered.
    This is only applicable when the write method is set to BigQueryIO.Write.Method.FILE_LOADS, and only when writing an unbounded PCollection.
    Every triggeringFrequency duration, a BigQuery load job will be generated for all the data written since the last load job. BigQuery has limits on how many load jobs can be triggered per day, so be careful not to set this duration too low, or you may exceed daily quota. Often this is set to 5 or 10 minutes to ensure that the project stays well under the BigQuery quota. See Quota Policy for more information about BigQuery quotas.
  - withNumFileShards
```
@Experimental
public BigQueryIO.Write<T> withNumFileShards(int numFileShards)
```
    Control how many file shards are written when using BigQuery load jobs. Applicable only when also setting withTriggeringFrequency(org.joda.time.Duration).
  - withCustomGcsTempLocation
```
public BigQueryIO.Write<T> withCustomGcsTempLocation(ValueProvider<java.lang.String> customGcsTempLocation)
```
    Provides a custom location on GCS for storing temporary files to be loaded via BigQuery batch load jobs. See "Usage with templates" in BigQueryIO documentation for discussion.
  - withExtendedErrorInfo
```
public BigQueryIO.Write<T> withExtendedErrorInfo()
```
    Enables extended error information by enabling WriteResult.getFailedInsertsWithErr()
    ATM this only works if using BigQueryIO.Write.Method.STREAMING_INSERTS. See withMethod(Method).
  - skipInvalidRows
```
public BigQueryIO.Write<T> skipInvalidRows()
```
    Insert all valid rows of a request, even if invalid rows exist. This is only applicable when the write method is set to BigQueryIO.Write.Method.STREAMING_INSERTS. The default value is false, which causes the entire request to fail if any invalid rows exist.
  - ignoreUnknownValues
```
public BigQueryIO.Write<T> ignoreUnknownValues()
```
    Accept rows that contain values that do not match the schema. The unknown values are ignored. Default is false, which treats unknown values as errors.
  - withKmsKey
```
public BigQueryIO.Write<T> withKmsKey(java.lang.String kmsKey)
```
  - optimizedWrites
```
@Experimental
public BigQueryIO.Write<T> optimizedWrites()
```
    If true, enables new codepaths that are expected to use less resources while writing to BigQuery. Not enabled by default in order to maintain backwards compatibility.
  - useBeamSchema
```
@Experimental
public BigQueryIO.Write<T> useBeamSchema()
```
    If true, then the BigQuery schema will be inferred from the input schema. If no formatFunction is set, then BigQueryIO will automatically turn the input records into TableRows that match the schema.
  - withTestServices
```
public BigQueryIO.Write<T> withTestServices(BigQueryServices testServices)
```
  - validate
```
public void validate(PipelineOptions pipelineOptions)
```
    Description copied from class: PTransform
    
    Called before running the Pipeline to verify this transform is fully and correctly specified.
    By default, does nothing.
    
    Overrides:
    
    validate in class PTransform<PCollection<T>,WriteResult>
  - expand
```
public WriteResult expand(PCollection<T> input)
```
    Description copied from class: PTransform
    
    Override this method to specify how this PTransform should be expanded on the given InputT.
    NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.
    Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
    
    Specified by:
    
    expand in class PTransform<PCollection<T>,WriteResult>
  - populateDisplayData
```
public void populateDisplayData(DisplayData.Builder builder)
```
    Description copied from class: PTransform
    
    Register display data for the given transform or component.
    populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.
    By default, does not register any display data. Implementors may override this method to provide their own display data.
    
    Specified by:
    
    populateDisplayData in interface HasDisplayData
    
    Overrides:
    
    populateDisplayData in class PTransform<PCollection<T>,WriteResult>
    
    Parameters:
    
    builder - The builder to populate with display data.
    
    See Also:
    
    HasDisplayData
  - getTable
```
@Nullable
public ValueProvider<TableReference> getTable()
```
    Returns the table reference, or null.

Class BigQueryIO.Write<T>

Nested Class Summary

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

Constructor Summary

Method Summary

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

Methods inherited from class java.lang.Object

Constructor Detail

Write

Method Detail

to

to

to

to

to

withFormatFunction

withSchema

withSchema

withJsonSchema

withJsonSchema

withSchemaFromView

withTimePartitioning

withTimePartitioning

withJsonTimePartitioning

withCreateDisposition

withWriteDisposition

withTableDescription

withFailedInsertRetryPolicy

withoutValidation

withMethod

withLoadJobProjectId

withLoadJobProjectId

withTriggeringFrequency

withNumFileShards

withCustomGcsTempLocation

withExtendedErrorInfo

skipInvalidRows

ignoreUnknownValues

withKmsKey

optimizedWrites

useBeamSchema

withTestServices

validate

expand

populateDisplayData

getTable