apache_beam.internal.gcp package

Submodules

apache_beam.internal.gcp.auth module

Dataflow credentials and authentication.

exception apache_beam.internal.gcp.auth.AuthenticationException[source]

Bases: apache_beam.utils.retry.PermanentException

apache_beam.internal.gcp.auth.get_service_credentials()[source]

For internal use only; no backwards-compatibility guarantees.

Get credentials to access Google services.

apache_beam.internal.gcp.auth.set_running_in_gce(worker_executing_project)[source]

For internal use only; no backwards-compatibility guarantees.

Informs the authentication library that we are running in GCE.

When we are running in GCE, we have the option of using the VM metadata credentials for authentication to Google services.

Parameters:worker_executing_project – The project running the workflow. This information comes from worker startup information.

apache_beam.internal.gcp.json_value module

JSON conversion utility functions.

apache_beam.internal.gcp.json_value.from_json_value(v)[source]

For internal use only; no backwards-compatibility guarantees.

Converts extra_types.JsonValue objects into Python objects.

Parameters:v – JsonValue object to be converted.
Returns:A Python object structured as values, lists, and dictionaries corresponding to JsonValue, JsonArray and JsonObject types.
Raises:TypeError – if the JsonValue object contains a type that is not supported.

The types supported are str, bool, list, dict, and None. The Dataflow API returns JsonValue(s) in many places and it is quite convenient to be able to convert these hierarchical objects to much simpler Python objects.

apache_beam.internal.gcp.json_value.get_typed_value_descriptor(obj)[source]

For internal use only; no backwards-compatibility guarantees.

Converts a basic type into a @type/value dictionary.

Parameters:obj – A basestring, bool, int, or float to be converted.
Returns:A dictionary containing the keys '@type‘ and ‘value’ with the value for the @type of appropriate type.
Raises:TypeError – if the Python object has a type that is not supported.
apache_beam.internal.gcp.json_value.to_json_value(obj, with_type=False)[source]

For internal use only; no backwards-compatibility guarantees.

Converts Python objects into extra_types.JsonValue objects.

Parameters:
  • obj – Python object to be converted. Can be ‘None’.
  • with_type – If true then the basic types (string, int, float, bool) will be wrapped in @type/value dictionaries. Otherwise the straight value is encoded into a JsonValue.
Returns:

A JsonValue object using JsonValue, JsonArray and JsonObject types for the corresponding values, lists, or dictionaries.
Raises:

TypeError – if the Python object contains a type that is not supported.

The types supported are str, bool, list, tuple, dict, and None. The Dataflow API requires JsonValue(s) in many places, and it is quite convenient to be able to specify these hierarchical objects using Python syntax.

Module contents

For internal use only; no backwards-compatibility guarantees.