Package org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryUtils

java.lang.Object
org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils
public class BigQueryUtils extends Object
Utility methods for BigQuery related operations.

  • Constructor Details

    • BigQueryUtils

      public BigQueryUtils()

  • Method Details

    • toTableSchema

      public static com.google.api.services.bigquery.model.TableSchema toTableSchema(Schema schema)
      Convert a Beam Schema to a BigQuery TableSchema.

    • fromTableSchema

      public static Schema fromTableSchema(com.google.api.services.bigquery.model.TableSchema tableSchema)
      Convert a BigQuery TableSchema to a Beam Schema.

    • fromTableSchema

      public static Schema fromTableSchema(com.google.api.services.bigquery.model.TableSchema tableSchema, BigQueryUtils.SchemaConversionOptions options)
      Convert a BigQuery TableSchema to a Beam Schema.

    • toGenericAvroSchema

      public static Schema toGenericAvroSchema(com.google.api.services.bigquery.model.TableSchema tableSchema)
      Convert a BigQuery TableSchema to Avro Schema.

    • fromGenericAvroSchema

      public static com.google.api.services.bigquery.model.TableSchema fromGenericAvroSchema(Schema schema)
      Convert an Avro Schema to a BigQuery TableSchema.

    • fromGenericAvroSchema

      public static com.google.api.services.bigquery.model.TableSchema fromGenericAvroSchema(Schema schema, Boolean useAvroLogicalTypes)
      Convert an Avro Schema to a BigQuery TableSchema.

    • toGenericAvroSchema

      public static Schema toGenericAvroSchema(com.google.api.services.bigquery.model.TableSchema tableSchema, Boolean useAvroLogicalTypes)
      Convert a BigQuery TableSchema to Avro Schema.

    • toGenericAvroSchema

      public static Schema toGenericAvroSchema(String schemaName, List<com.google.api.services.bigquery.model.TableFieldSchema> fieldSchemas)
      Convert a list of BigQuery TableFieldSchema to Avro Schema.

    • toGenericAvroSchema

      public static Schema toGenericAvroSchema(String schemaName, List<com.google.api.services.bigquery.model.TableFieldSchema> fieldSchemas, Boolean useAvroLogicalTypes)
      Convert a list of BigQuery TableFieldSchema to Avro Schema.

    • tableRowToBeamRow

      public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<com.google.api.services.bigquery.model.TableRow> tableRowToBeamRow()

    • tableRowFromBeamRow

      public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<com.google.api.services.bigquery.model.TableRow> tableRowFromBeamRow()

    • toTableRow

      public static SerializableFunction<Row,com.google.api.services.bigquery.model.TableRow> toTableRow()
      Convert a Beam Row to a BigQuery TableRow.

    • toTableRow

      public static <T> SerializableFunction<T,com.google.api.services.bigquery.model.TableRow> toTableRow(SerializableFunction<T,Row> toRow)
      Convert a Beam schema type to a BigQuery TableRow.

    • toBeamRow

      public static Row toBeamRow(GenericRecord record, Schema schema, BigQueryUtils.ConversionOptions options)

    • convertGenericRecordToTableRow

      @Deprecated public static com.google.api.services.bigquery.model.TableRow convertGenericRecordToTableRow(GenericRecord record, com.google.api.services.bigquery.model.TableSchema tableSchema)
      Deprecated.
      use convertGenericRecordToTableRow(GenericRecord)
      Convert generic record to Bq TableRow.

    • convertGenericRecordToTableRow

      public static com.google.api.services.bigquery.model.TableRow convertGenericRecordToTableRow(GenericRecord record)
      Convert generic record to Bq TableRow.

    • toTableRow

      public static com.google.api.services.bigquery.model.TableRow toTableRow(Row row)
      Convert a Beam Row to a BigQuery TableRow.

    • toBeamRow

      public static Row toBeamRow(Schema rowSchema, com.google.api.services.bigquery.model.TableRow jsonBqRow)
      Tries to convert a JSON TableRow from BigQuery into a Beam Row.

      Only supports basic types and arrays. Doesn't support date types or structs.

    • toBeamRow

      public static Row toBeamRow(Schema rowSchema, com.google.api.services.bigquery.model.TableSchema bqSchema, com.google.api.services.bigquery.model.TableRow jsonBqRow)
      Tries to parse the JSON TableRow from BigQuery.

      Only supports basic types and arrays. Doesn't support date types.

    • convertAvroFormat

      public static @Nullable Object convertAvroFormat(Schema.FieldType beamFieldType, @Nullable Object avroValue, BigQueryUtils.ConversionOptions options)
      Tries to convert an Avro decoded value to a Beam field value based on the target type of the Beam field.

      For the Avro formats of BigQuery types, see https://cloud.google.com/bigquery/docs/exporting-data#avro_export_details and https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions

    • toTableReference

      public static @Nullable com.google.api.services.bigquery.model.TableReference toTableReference(String fullTableId)
      Returns a TableReference by parsing the fullTableId. If it cannot be parsed properly null is returned.
      Parameters:
      fullTableId - - Is one of the two forms commonly used to refer to bigquery tables in the beam codebase:
      • projects/{project_id}/datasets/{dataset_id}/tables/{table_id}
      • myproject:mydataset.mytable
      • myproject.mydataset.mytable

    • toTableSpec

      public static @Nullable String toTableSpec(com.google.api.services.bigquery.model.TableReference tableReference)
      Returns a String representation of the table destination in the form: `myproject.mydataset.mytable`.
      Parameters:
      tableReference - - a BigQueryTableIdentifier that may or may not include the project.

    • readCallMetric

      public static @Nullable org.apache.beam.runners.core.metrics.ServiceCallMetric readCallMetric(@Nullable com.google.api.services.bigquery.model.TableReference tableReference)
      Returns a ServiceCallMetric for recording statuses for all BQ API responses related to reading elements directly from BigQuery in a process-wide metric. Such as: calls to readRows, splitReadStream, createReadSession.
      Parameters:
      tableReference - - The table being read from. Can be a temporary BQ table used to read from a SQL query.

    • writeCallMetric

      public static org.apache.beam.runners.core.metrics.ServiceCallMetric writeCallMetric(com.google.api.services.bigquery.model.TableReference tableReference)
      Returns a ServiceCallMetric for recording statuses for all BQ responses related to writing elements directly to BigQuery in a process-wide metric. Such as: insertAll.
      Parameters:
      tableReference - - The table being written to.