apache_beam.utils.secret module

Interface and implementations for Secret providers in Apache Beam.

class apache_beam.utils.secret.Secret[source]

Bases: ABC

A secret management class used for handling sensitive data.

This class provides a generic interface for secret management. Implementations of this class should handle fetching secrets from a secret management system.

get_str(cacheSecret: bool = False) → str[source]

Retrieve secret value as string.

Parameters:

cacheSecret – If True, caches secret value in memory after first fetch.

Returns:

The retrieved secret value as string.

get_bytes(cacheSecret: bool = False) → bytes[source]

Retrieve secret value as bytes.

Parameters:

cacheSecret – If True, caches secret value in memory after first fetch.

Returns:

The retrieved secret value as bytes.

abstract get_secret_bytes() → bytes[source]

Returns the secret as a byte string.

static generate_secret_bytes() → bytes[source]

Generates a new secret key using Fernet.

apply_pipeline_options(options: Any) → None[source]

Configures this secret from pipeline options.

Called by from_json() and parse_secret_option() when pipeline options are given. The default implementation does nothing; secrets that need settings from pipeline options (e.g. AwsSecret) override it.

Parameters:

options – PipelineOptions, or a dict of option names to values.

classmethod parse_secret_option(secret: str, options: Any | None = None) → Secret[source]

Parses a secret string and returns the appropriate secret type.

The secret string should be formatted like: ‘type:<secret_type>;<secret_param>:<value>’

Only the first ‘:’ of each parameter separates the name from the value, so values may contain ‘:’ (e.g. ARNs or URLs), but not ‘;’.

For example, ‘type:GcpSecret;version_name:my_secret/versions/latest’ would return a GcpSecret initialized with ‘my_secret/versions/latest’.

Parameters:
  • secret – The secret string.

  • options – Optional pipeline options, passed to apply_pipeline_options() of the returned secret.

classmethod from_json(spec: str, secret_manager: str | None = None, options: Any | None = None) → Secret[source]

Return a Secret instance based on secret_manager provider and secret specification.

Parameters:
  • spec – Secret string (raw secret or JSON specification string).

  • secret_manager – Secret manager string (e.g. ‘GoogleCloudSecretManager’).

  • options – Optional pipeline options (PipelineOptions or a dict), passed to apply_pipeline_options() of the returned secret. For example, AwsSecret uses them to get AWS credentials from AwsOptions.

Returns:

An instance of Secret.

class apache_beam.utils.secret.RawSecret(secret: str | bytes)[source]

Bases: Secret

Secret implementation wrapping a raw secret string or bytes directly.

get_secret_bytes() → bytes[source]
class apache_beam.utils.secret.GcpSecret(version_name: str)[source]

Bases: Secret

A secret manager implementation that retrieves secrets from Google Cloud Secret Manager.

Initializes a GcpSecret object.

Parameters:

version_name – The full version name of the secret in Google Cloud Secret Manager. For example: projects/<id>/secrets/<secret_name>/versions/1. For more info, see https://cloud.google.com/python/docs/reference/secretmanager/latest/google.cloud.secretmanager_v1beta1.services.secret_manager_service.SecretManagerServiceClient#google_cloud_secretmanager_v1beta1_services_secret_manager_service_SecretManagerServiceClient_access_secret_version

classmethod from_dict(spec_dict: dict[str, str]) → GcpSecret[source]

Initialize GcpSecret from a dictionary specification.

get_secret_bytes() → bytes[source]
class apache_beam.utils.secret.AwsSecret(secret_id: str, version_id: str | None = None, version_stage: str | None = None, **client_settings: Any)[source]

Bases: Secret

A secret manager implementation that retrieves secrets from AWS Secrets Manager.

AWS credentials and connection settings (region, endpoint, SSL) come from the --aws_* defaults in AwsOptions when pipeline options are given (see apply_pipeline_options()), and otherwise from boto3’s defaults (e.g. environment variables, ~/.aws, or an IAM role). Connection settings can be overridden per secret with keyword arguments, e.g. AwsSecret('my-secret', region_name='us-west-2'), or in the secret spec, e.g. type:awssecret;name:my-secret;region_name:us-west-2.

Initializes an AwsSecret object.

Parameters:
  • secret_id – The ARN or name of the secret in AWS Secrets Manager.

  • version_id – The unique identifier of the version of the secret.

  • version_stage – The staging label of the version of the secret (e.g. ‘AWSCURRENT’).

  • **client_settings – AWS client connection settings that override the --aws_* pipeline options for this secret. Names are AwsOptions setting names without the aws_ prefix (e.g. region_name, endpoint_url, verify, disable_ssl). Credentials cannot be set here; use the --aws_* pipeline options instead.

classmethod from_dict(spec_dict: dict[str, str]) → AwsSecret[source]

Initialize AwsSecret from a dictionary specification.

Keys other than the secret identifiers are AWS client connection settings (see the **client_settings keyword arguments of __init__()).

apply_pipeline_options(options: Any) → None[source]

Uses AWS credentials and connection settings from AwsOptions.

Only the (unresolved) option values are stored; credential secret specs are resolved when the secret is fetched, e.g. on workers. Settings in client_settings still take precedence.

Parameters:

options – PipelineOptions, or a dict of option names to values.

get_secret_bytes() → bytes[source]
class apache_beam.utils.secret.GcpHsmGeneratedSecret(project_id: str, location_id: str, key_ring_id: str, key_id: str, job_name: str)[source]

Bases: Secret

A secret manager implementation that generates a secret using a GCP HSM key and stores it in Google Cloud Secret Manager. If the secret already exists, it will be retrieved.

Initializes a GcpHsmGeneratedSecret object.

Parameters:
  • project_id – The GCP project ID.

  • location_id – The GCP location ID for the HSM key.

  • key_ring_id – The ID of the KMS key ring.

  • key_id – The ID of the KMS key.

  • job_name – The name of the job, used to generate a unique secret name.

classmethod from_dict(spec_dict: dict[str, str]) → GcpHsmGeneratedSecret[source]

Initialize GcpHsmGeneratedSecret from a dictionary specification.

get_secret_bytes() → bytes[source]

Retrieves the secret bytes.

If the secret version already exists in Secret Manager, it is retrieved. Otherwise, a new secret and version are created. The new secret is generated using the HSM key.

Returns:

The secret as a byte string.

generate_dek(dek_size: int = 32) → bytes[source]

Generates a new Data Encryption Key (DEK) using an HSM-backed key.

This function follows a key derivation process that incorporates entropy from the HSM-backed key into the nonce used for key derivation.

Parameters:

dek_size – The size of the DEK to generate.

Returns:

A new DEK of the specified size, url-safe base64-encoded.