Package org.apache.beam.sdk.io.gcp.bigquery
package org.apache.beam.sdk.io.gcp.bigquery
Defines transforms for reading and writing from Google BigQuery.
- See Also:
-
ClassDescriptionUtility methods for converting Avro
GenericRecordobjects to dynamic protocol message, for use with the Storage write API.Utility methods for converting BeamRowobjects to dynamic protocol message, for use with the Storage write API.ACoderProviderRegistrarfor standard types used withBigQueryIO.Configuration for reading from BigQuery.An implementation ofTypedSchemaTransformProviderfor BigQuery read jobs configured usingBigQueryExportReadSchemaTransformConfiguration.An implementation ofSchemaTransformfor BigQuery read jobs configured usingBigQueryExportReadSchemaTransformConfiguration.A set of helper functions and classes used byBigQueryIO.Model definition for BigQueryInsertError.ACoderthat encodes BigQueryBigQueryInsertErrorobjects.PTransforms for reading and writing BigQuery tables.Implementation ofBigQueryIO.read().Implementation ofBigQueryIO.read(SerializableFunction).Determines the method used to read data from BigQuery.An enumeration type for the priority of a query.Implementation ofBigQueryIO.write().An enumeration type for the BigQuery create disposition strings.Determines the method used to insert data in BigQuery.An enumeration type for the BigQuery schema update options strings.An enumeration type for the BigQuery write disposition strings.Properties needed when using Google BigQuery with the Apache Beam SDK.An implementation ofSchemaIOProviderfor reading and writing to BigQuery withBigQueryIO.Exception to signal that BigQuery schema retrieval failed.An interface for real, mock, or fake implementations of Cloud BigQuery services.Container for reading data from streaming endpoints.An interface to get, create and delete Cloud BigQuery datasets and tables.An interface for the Cloud BigQuery load service.An interface representing a client object for making calls to the BigQuery Storage API.An interface for appending records to a Storage API write stream.An interface to get, create and flush Cloud BigQuery STORAGE API write streams.An implementation ofBigQueryServicesthat actually communicates with the cloud BigQuery service.Helper class to create perworker metrics for BigQuery Sink stages.ASourcerepresenting reading from a table.Utility methods for BigQuery related operations.Options for how to convert BigQuery data to Beam data.Builder forBigQueryUtils.ConversionOptions.Controls whether to truncate timestamps to millisecond precision lossily, or to crash when truncation would result.Options for how to convert BigQuery schemas to Beam schemas.Builder forBigQueryUtils.SchemaConversionOptions.Encoder for TIME and DATETIME values, according to civil_time encoding.CreateTables<DestinationT,ElementT> Creates any tables needed before performing streaming writes to the tables.DynamicDestinations<T,DestinationT> This class provides the most general way of specifying dynamic BigQuery table destinations.ErrorContainer interface.A retry policy for streaming BigQuery inserts.Contains information about a failed insert.PrepareWrite<InputT,DestinationT extends @NonNull Object, OutputT> Prepare an inputPCollectionfor writing to BigQuery.This transforms turns a side input into a singleton PCollection that can be used as the main input for another transform.A convenience class for applying row updates to BigQuery usingBigQueryIO.applyRowMutations().This class indicates how to apply a row update to BigQuery.A wrapper for aGenericRecordand theTableSchemarepresenting the schema of the table (or query) it was generated from.Constants and variables for CDC support.StorageApiConvertMessages<DestinationT,ElementT> A transform that converts messages to protocol buffers in preparation for writing to BigQuery.StorageApiDynamicDestinationsTableRow<T,DestinationT extends @NonNull Object> This DoFn flushes and optionally (if requested) finalizes Storage API streams.StorageApiLoads<DestinationT,ElementT> ThisPTransformmanages loads into BigQuery using the Storage API.Class used to wrap elements being sent to the Storage API sinks.StorageApiWriteRecordsInconsistent<DestinationT,ElementT> A transform to write sharded records to BigQuery using the Storage API.A transform to write sharded records to BigQuery using the Storage API (Streaming).StorageApiWriteUnshardedRecords<DestinationT,ElementT> Write records to the Storage API using a standard batch approach.StreamingInserts<DestinationT,ElementT> PTransform that performs streaming BigQuery write.Stores and exports metrics for a batch of Streaming Inserts RPCs.No-op implementation ofStreamingInsertsResults.Metrics of a batch of InsertAll RPCs.StreamingWriteTables<ElementT>This transform takes in key-value pairs ofTableRowentries and theTableDestinationit should be written to.Encapsulates a BigQuery table destination.A coder forTableDestinationobjects.ACoderforTableDestinationthat includes time partitioning information.ACoderforTableDestinationthat includes time partitioning and clustering information.Utility methods for converting JSONTableRowobjects to dynamic protocol message, for use with the Storage write API.An updatable cache for table schemas.Helper utilities for handling schema-update responses.Test rule which creates a new table with specified schema, with randomized name and exposes few APIs to work with it.Interface to implement a polling assertion.UpdateSchemaDestination<DestinationT>Update destination schema based on data that is about to be copied into it.The result of aBigQueryIO.Writetransform.