apache_beam.io.gcp.bigquery_schema_tools module
Tools used tool work with Schema types in the context of BigQuery. Classes, constants and functions in this file are experimental and have no backwards compatibility guarantees. NOTHING IN THIS FILE HAS BACKWARDS COMPATIBILITY GUARANTEES.
- apache_beam.io.gcp.bigquery_schema_tools.generate_user_type_from_bq_schema(the_table_schema, selected_fields: TableSchema | None = None, type_overrides=None) type[source]
Convert a schema of type TableSchema into a pcollection element.
- Parameters:
the_table_schema – A BQ schema of type TableSchema
selected_fields – if not None, the subset of fields to consider
type_overrides – Optional mapping of BigQuery type names (uppercase) to Python types. These override the default mappings in BIG_QUERY_TO_PYTHON_TYPES. For example:
{'DATE': datetime.date, 'JSON': dict}
- Returns:
type that can be used to work with pCollections.
- Return type:
- apache_beam.io.gcp.bigquery_schema_tools.bq_field_to_type(field, mode, type_overrides=None)[source]
Convert a BigQuery field type and mode to a Python type hint.
- Parameters:
field – The BigQuery type name (e.g., ‘STRING’, ‘DATE’).
mode – The field mode (‘NULLABLE’, ‘REPEATED’, ‘REQUIRED’).
type_overrides – Optional mapping of BigQuery type names (uppercase) to Python types. These override the default mappings.
- Returns:
The corresponding Python type hint.
- apache_beam.io.gcp.bigquery_schema_tools.convert_to_usertype(table_schema, selected_fields=None, type_overrides=None)[source]
Convert a BigQuery table schema to a user type.
- Parameters:
table_schema – A BQ schema of type TableSchema
selected_fields – if not None, the subset of fields to consider
type_overrides – Optional mapping of BigQuery type names (uppercase) to Python types.
- Returns:
A ParDo transform that converts dictionaries to the user type.