org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryUtils

java.lang.Object

org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils

public class BigQueryUtils
extends java.lang.Object

Utility methods for BigQuery related operations.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigQueryUtils.ConversionOptions Options for how to convert BigQuery data to Beam data.
static class	BigQueryUtils.SchemaConversionOptions Options for how to convert BigQuery schemas to Beam schemas.

Constructor Summary

Constructors

Constructor and Description
BigQueryUtils()

Method Summary

Modifier and Type	Method and Description
static java.lang.Object	convertAvroFormat(Schema.FieldType beamFieldType, java.lang.Object avroValue, BigQueryUtils.ConversionOptions options) Tries to convert an Avro decoded value to a Beam field value based on the target type of the Beam field.
static TableRow	convertGenericRecordToTableRow(GenericRecord record) Convert generic record to Bq TableRow.
static TableRow	convertGenericRecordToTableRow(GenericRecord record, TableSchema tableSchema) Deprecated. use convertGenericRecordToTableRow(GenericRecord)
static Schema	fromTableSchema(TableSchema tableSchema) Convert a BigQuery TableSchema to a Beam Schema.
static Schema	fromTableSchema(TableSchema tableSchema, BigQueryUtils.SchemaConversionOptions options) Convert a BigQuery TableSchema to a Beam Schema.
static @Nullable org.apache.beam.runners.core.metrics.ServiceCallMetric	readCallMetric(@Nullable TableReference tableReference)
static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<TableRow>	tableRowFromBeamRow()
static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<TableRow>	tableRowToBeamRow()
static Row	toBeamRow(GenericRecord record, Schema schema, BigQueryUtils.ConversionOptions options)
static Row	toBeamRow(Schema rowSchema, TableRow jsonBqRow) Tries to convert a JSON TableRow from BigQuery into a Beam Row.
static Row	toBeamRow(Schema rowSchema, TableSchema bqSchema, TableRow jsonBqRow) Tries to parse the JSON TableRow from BigQuery.
static Schema	toGenericAvroSchema(java.lang.String schemaName, java.util.List<TableFieldSchema> fieldSchemas) Convert a list of BigQuery TableFieldSchema to Avro Schema.
static Schema	toGenericAvroSchema(java.lang.String schemaName, java.util.List<TableFieldSchema> fieldSchemas, java.lang.Boolean useAvroLogicalTypes) Convert a list of BigQuery TableFieldSchema to Avro Schema.
static Schema	toGenericAvroSchema(TableSchema tableSchema) Convert a list of BigQuery TableSchema to Avro Schema.
static Schema	toGenericAvroSchema(TableSchema tableSchema, java.lang.Boolean useAvroLogicalTypes) Convert a list of BigQuery TableSchema to Avro Schema.
static @Nullable TableReference	toTableReference(java.lang.String fullTableId)
static SerializableFunction<Row,TableRow>	toTableRow() Convert a Beam Row to a BigQuery TableRow.
static TableRow	toTableRow(Row row) Convert a Beam Row to a BigQuery TableRow.
static <T> SerializableFunction<T,TableRow>	toTableRow(SerializableFunction<T,Row> toRow) Convert a Beam schema type to a BigQuery TableRow.
static TableSchema	toTableSchema(Schema schema) Convert a Beam Schema to a BigQuery TableSchema.
static @Nullable java.lang.String	toTableSpec(TableReference tableReference)
static org.apache.beam.runners.core.metrics.ServiceCallMetric	writeCallMetric(TableReference tableReference)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryUtils

public BigQueryUtils()

Method Detail

toTableSchema

public static TableSchema toTableSchema(Schema schema)

Convert a Beam Schema to a BigQuery TableSchema.

fromTableSchema

public static Schema fromTableSchema(TableSchema tableSchema)

Convert a BigQuery TableSchema to a Beam Schema.

fromTableSchema

public static Schema fromTableSchema(TableSchema tableSchema,
                                     BigQueryUtils.SchemaConversionOptions options)

Convert a BigQuery TableSchema to a Beam Schema.

toGenericAvroSchema

public static Schema toGenericAvroSchema(TableSchema tableSchema)

Convert a list of BigQuery TableSchema to Avro Schema.

toGenericAvroSchema

public static Schema toGenericAvroSchema(TableSchema tableSchema,
                                         java.lang.Boolean useAvroLogicalTypes)

Convert a list of BigQuery TableSchema to Avro Schema.

toGenericAvroSchema

public static Schema toGenericAvroSchema(java.lang.String schemaName,
                                         java.util.List<TableFieldSchema> fieldSchemas)

Convert a list of BigQuery TableFieldSchema to Avro Schema.

toGenericAvroSchema

public static Schema toGenericAvroSchema(java.lang.String schemaName,
                                         java.util.List<TableFieldSchema> fieldSchemas,
                                         java.lang.Boolean useAvroLogicalTypes)

Convert a list of BigQuery TableFieldSchema to Avro Schema.

tableRowToBeamRow

public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<TableRow> tableRowToBeamRow()

tableRowFromBeamRow

public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<TableRow> tableRowFromBeamRow()

toTableRow

public static SerializableFunction<Row,TableRow> toTableRow()

Convert a Beam Row to a BigQuery TableRow.

toTableRow

public static <T> SerializableFunction<T,TableRow> toTableRow(SerializableFunction<T,Row> toRow)

Convert a Beam schema type to a BigQuery TableRow.

toBeamRow

public static Row toBeamRow(GenericRecord record,
                            Schema schema,
                            BigQueryUtils.ConversionOptions options)

convertGenericRecordToTableRow

@Deprecated
public static TableRow convertGenericRecordToTableRow(GenericRecord record,
                                                                  TableSchema tableSchema)

Deprecated. use convertGenericRecordToTableRow(GenericRecord)

Convert generic record to Bq TableRow.

convertGenericRecordToTableRow

public static TableRow convertGenericRecordToTableRow(GenericRecord record)

Convert generic record to Bq TableRow.

toTableRow

public static TableRow toTableRow(Row row)

Convert a Beam Row to a BigQuery TableRow.

toBeamRow

public static Row toBeamRow(Schema rowSchema,
                            TableRow jsonBqRow)

Tries to convert a JSON TableRow from BigQuery into a Beam Row.

Only supports basic types and arrays. Doesn't support date types or structs.

toBeamRow

public static Row toBeamRow(Schema rowSchema,
                            TableSchema bqSchema,
                            TableRow jsonBqRow)

Tries to parse the JSON TableRow from BigQuery.

Only supports basic types and arrays. Doesn't support date types.

convertAvroFormat

public static java.lang.Object convertAvroFormat(Schema.FieldType beamFieldType,
                                                 java.lang.Object avroValue,
                                                 BigQueryUtils.ConversionOptions options)

Tries to convert an Avro decoded value to a Beam field value based on the target type of the Beam field.

For the Avro formats of BigQuery types, see https://cloud.google.com/bigquery/docs/exporting-data#avro_export_details and https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions

toTableReference

public static @Nullable TableReference toTableReference(java.lang.String fullTableId)

Parameters:

fullTableId - - Is one of the two forms commonly used to refer to bigquery tables in the beam codebase:

• projects/{project_id}/datasets/{dataset_id}/tables/{table_id}
• myproject:mydataset.mytable
• myproject.mydataset.mytable

Returns:

a BigQueryTableIdentifier by parsing the fullTableId. If it cannot be parsed properly null is returned.

toTableSpec

public static @Nullable java.lang.String toTableSpec(TableReference tableReference)

Parameters:

tableReference - - a BigQueryTableIdentifier that may or may not include the project.

Returns:

a String representation of the table destination in the form: `myproject.mydataset.mytable`

readCallMetric

public static @Nullable org.apache.beam.runners.core.metrics.ServiceCallMetric readCallMetric(@Nullable TableReference tableReference)

Parameters:

tableReference - - The table being read from. Can be a temporary BQ table used to read from a SQL query.

Returns:

a ServiceCallMetric for recording statuses for all BQ API responses related to reading elements directly from BigQuery in a process-wide metric. Such as: calls to readRows, splitReadStream, createReadSession.

writeCallMetric

public static org.apache.beam.runners.core.metrics.ServiceCallMetric writeCallMetric(TableReference tableReference)

Parameters:

tableReference - - The table being written to.

Returns:

a ServiceCallMetric for recording statuses for all BQ responses related to writing elements directly to BigQuery in a process-wide metric. Such as: insertAll.