Package org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryUtils

java.lang.Object
org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils
public class BigQueryUtils extends Object
Utility methods for BigQuery related operations.

  • Constructor Details

    • BigQueryUtils

      public BigQueryUtils()

  • Method Details

    • toTableSchema

      public static TableSchema toTableSchema(Schema schema)
      Convert a Beam Schema to a BigQuery TableSchema.

    • fromTableSchema

      public static Schema fromTableSchema(TableSchema tableSchema)
      Convert a BigQuery TableSchema to a Beam Schema.

    • fromTableSchema

      public static Schema fromTableSchema(TableSchema tableSchema, BigQueryUtils.SchemaConversionOptions options)
      Convert a BigQuery TableSchema to a Beam Schema.

    • toGenericAvroSchema

      public static Schema toGenericAvroSchema(TableSchema tableSchema)
      Convert a BigQuery TableSchema to Avro Schema.

    • fromGenericAvroSchema

      public static TableSchema fromGenericAvroSchema(Schema schema)
      Convert an Avro Schema to a BigQuery TableSchema.

    • fromGenericAvroSchema

      public static TableSchema fromGenericAvroSchema(Schema schema, Boolean useAvroLogicalTypes)
      Convert an Avro Schema to a BigQuery TableSchema.

    • toGenericAvroSchema

      public static Schema toGenericAvroSchema(TableSchema tableSchema, Boolean useAvroLogicalTypes)
      Convert a BigQuery TableSchema to Avro Schema.

    • toGenericAvroSchema

      public static Schema toGenericAvroSchema(String schemaName, List<TableFieldSchema> fieldSchemas)
      Convert a list of BigQuery TableFieldSchema to Avro Schema.

    • toGenericAvroSchema

      public static Schema toGenericAvroSchema(String schemaName, List<TableFieldSchema> fieldSchemas, Boolean useAvroLogicalTypes)
      Convert a list of BigQuery TableFieldSchema to Avro Schema.

    • tableRowToBeamRow

      public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<TableRow> tableRowToBeamRow()

    • tableRowFromBeamRow

      public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<TableRow> tableRowFromBeamRow()

    • toTableRow

      public static SerializableFunction<Row,TableRow> toTableRow()
      Convert a Beam Row to a BigQuery TableRow.

    • toTableRow

      public static <T> SerializableFunction<T,TableRow> toTableRow(SerializableFunction<T,Row> toRow)
      Convert a Beam schema type to a BigQuery TableRow.

    • toBeamRow

      public static Row toBeamRow(GenericRecord record, Schema schema, BigQueryUtils.ConversionOptions options)

    • convertGenericRecordToTableRow

      @Deprecated public static TableRow convertGenericRecordToTableRow(GenericRecord record, TableSchema tableSchema)
      Deprecated.
      use convertGenericRecordToTableRow(GenericRecord)
      Convert generic record to Bq TableRow.

    • convertGenericRecordToTableRow

      public static TableRow convertGenericRecordToTableRow(GenericRecord record)
      Convert generic record to Bq TableRow.

    • toTableRow

      public static TableRow toTableRow(Row row)
      Convert a Beam Row to a BigQuery TableRow.

    • toBeamRow

      public static Row toBeamRow(Schema rowSchema, TableRow jsonBqRow)
      Tries to convert a JSON TableRow from BigQuery into a Beam Row.

      Only supports basic types and arrays. Doesn't support date types or structs.

    • toBeamRow

      public static Row toBeamRow(Schema rowSchema, TableSchema bqSchema, TableRow jsonBqRow)
      Tries to parse the JSON TableRow from BigQuery.

      Only supports basic types and arrays. Doesn't support date types.

    • convertAvroFormat

      public static Object convertAvroFormat(Schema.FieldType beamFieldType, Object avroValue, BigQueryUtils.ConversionOptions options)
      Tries to convert an Avro decoded value to a Beam field value based on the target type of the Beam field.

      For the Avro formats of BigQuery types, see https://cloud.google.com/bigquery/docs/exporting-data#avro_export_details and https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions

    • toTableReference

      public static @Nullable TableReference toTableReference(String fullTableId)
      Parameters:
      fullTableId - - Is one of the two forms commonly used to refer to bigquery tables in the beam codebase:
      • projects/{project_id}/datasets/{dataset_id}/tables/{table_id}
      • myproject:mydataset.mytable
      • myproject.mydataset.mytable
      Returns:
      a BigQueryTableIdentifier by parsing the fullTableId. If it cannot be parsed properly null is returned.

    • toTableSpec

      public static @Nullable String toTableSpec(TableReference tableReference)
      Parameters:
      tableReference - - a BigQueryTableIdentifier that may or may not include the project.
      Returns:
      a String representation of the table destination in the form: `myproject.mydataset.mytable`

    • readCallMetric

      public static @Nullable org.apache.beam.runners.core.metrics.ServiceCallMetric readCallMetric(@Nullable TableReference tableReference)
      Parameters:
      tableReference - - The table being read from. Can be a temporary BQ table used to read from a SQL query.
      Returns:
      a ServiceCallMetric for recording statuses for all BQ API responses related to reading elements directly from BigQuery in a process-wide metric. Such as: calls to readRows, splitReadStream, createReadSession.

    • writeCallMetric

      public static org.apache.beam.runners.core.metrics.ServiceCallMetric writeCallMetric(TableReference tableReference)
      Parameters:
      tableReference - - The table being written to.
      Returns:
      a ServiceCallMetric for recording statuses for all BQ responses related to writing elements directly to BigQuery in a process-wide metric. Such as: insertAll.