apache_beam.utils.secret module

Interface and implementations for Secret providers in Apache Beam.

class apache_beam.utils.secret.Secret[source]

Bases: ABC

A secret management class used for handling sensitive data.

This class provides a generic interface for secret management. Implementations of this class should handle fetching secrets from a secret management system.

get_str(cacheSecret: bool = False) → str[source]

Retrieve secret value as string.

Parameters:

cacheSecret – If True, caches secret value in memory after first fetch.

Returns:

The retrieved secret value as string.

get_bytes(cacheSecret: bool = False) → bytes[source]

Retrieve secret value as bytes.

Parameters:

cacheSecret – If True, caches secret value in memory after first fetch.

Returns:

The retrieved secret value as bytes.

abstract get_secret_bytes() → bytes[source]

Returns the secret as a byte string.

static generate_secret_bytes() → bytes[source]

Generates a new secret key using Fernet.

classmethod parse_secret_option(secret: str) → Secret[source]

Parses a secret string and returns the appropriate secret type.

The secret string should be formatted like: ‘type:<secret_type>;<secret_param>:<value>’

For example, ‘type:GcpSecret;version_name:my_secret/versions/latest’ would return a GcpSecret initialized with ‘my_secret/versions/latest’.

classmethod from_json(spec: str, secret_manager: str | None = None) → Secret[source]

Return a Secret instance based on secret_manager provider and secret specification.

Parameters:

  • spec – Secret string (raw secret or JSON specification string).

  • secret_manager – Secret manager string (e.g. ‘GoogleCloudSecretManager’).

Returns:

An instance of Secret.

class apache_beam.utils.secret.RawSecret(secret: str | bytes)[source]

Bases: Secret

Secret implementation wrapping a raw secret string or bytes directly.

get_secret_bytes() → bytes[source]
class apache_beam.utils.secret.GcpSecret(version_name: str)[source]

Bases: Secret

A secret manager implementation that retrieves secrets from Google Cloud Secret Manager.

Initializes a GcpSecret object.

Parameters:

version_name – The full version name of the secret in Google Cloud Secret Manager. For example: projects/<id>/secrets/<secret_name>/versions/1. For more info, see https://cloud.google.com/python/docs/reference/secretmanager/latest/google.cloud.secretmanager_v1beta1.services.secret_manager_service.SecretManagerServiceClient#google_cloud_secretmanager_v1beta1_services_secret_manager_service_SecretManagerServiceClient_access_secret_version

classmethod from_dict(spec_dict: Dict[str, str]) → GcpSecret[source]

Initialize GcpSecret from a dictionary specification.

get_secret_bytes() → bytes[source]
class apache_beam.utils.secret.GcpHsmGeneratedSecret(project_id: str, location_id: str, key_ring_id: str, key_id: str, job_name: str)[source]

Bases: Secret

A secret manager implementation that generates a secret using a GCP HSM key and stores it in Google Cloud Secret Manager. If the secret already exists, it will be retrieved.

Initializes a GcpHsmGeneratedSecret object.

Parameters:

  • project_id – The GCP project ID.

  • location_id – The GCP location ID for the HSM key.

  • key_ring_id – The ID of the KMS key ring.

  • key_id – The ID of the KMS key.

  • job_name – The name of the job, used to generate a unique secret name.

classmethod from_dict(spec_dict: Dict[str, str]) → GcpHsmGeneratedSecret[source]

Initialize GcpHsmGeneratedSecret from a dictionary specification.

get_secret_bytes() → bytes[source]

Retrieves the secret bytes.

If the secret version already exists in Secret Manager, it is retrieved. Otherwise, a new secret and version are created. The new secret is generated using the HSM key.

Returns:

The secret as a byte string.

generate_dek(dek_size: int = 32) → bytes[source]

Generates a new Data Encryption Key (DEK) using an HSM-backed key.

This function follows a key derivation process that incorporates entropy from the HSM-backed key into the nonce used for key derivation.

Parameters:

dek_size – The size of the DEK to generate.

Returns:

A new DEK of the specified size, url-safe base64-encoded.