apache_beam.utils.secret module
Interface and implementations for Secret providers in Apache Beam.
- class apache_beam.utils.secret.Secret[source]
Bases:
ABCA secret management class used for handling sensitive data.
This class provides a generic interface for secret management. Implementations of this class should handle fetching secrets from a secret management system.
- get_str(cacheSecret: bool = False) str[source]
Retrieve secret value as string.
- Parameters:
cacheSecret – If True, caches secret value in memory after first fetch.
- Returns:
The retrieved secret value as string.
- get_bytes(cacheSecret: bool = False) bytes[source]
Retrieve secret value as bytes.
- Parameters:
cacheSecret – If True, caches secret value in memory after first fetch.
- Returns:
The retrieved secret value as bytes.
- apply_pipeline_options(options: Any) None[source]
Configures this secret from pipeline options.
Called by
from_json()andparse_secret_option()when pipeline options are given. The default implementation does nothing; secrets that need settings from pipeline options (e.g.AwsSecret) override it.- Parameters:
options – PipelineOptions, or a dict of option names to values.
- classmethod parse_secret_option(secret: str, options: Any | None = None) Secret[source]
Parses a secret string and returns the appropriate secret type.
The secret string should be formatted like: ‘type:<secret_type>;<secret_param>:<value>’
Only the first ‘:’ of each parameter separates the name from the value, so values may contain ‘:’ (e.g. ARNs or URLs), but not ‘;’.
For example, ‘type:GcpSecret;version_name:my_secret/versions/latest’ would return a GcpSecret initialized with ‘my_secret/versions/latest’.
- Parameters:
secret – The secret string.
options – Optional pipeline options, passed to
apply_pipeline_options()of the returned secret.
- classmethod from_json(spec: str, secret_manager: str | None = None, options: Any | None = None) Secret[source]
Return a Secret instance based on secret_manager provider and secret specification.
- Parameters:
spec – Secret string (raw secret or JSON specification string).
secret_manager – Secret manager string (e.g. ‘GoogleCloudSecretManager’).
options – Optional pipeline options (PipelineOptions or a dict), passed to
apply_pipeline_options()of the returned secret. For example, AwsSecret uses them to get AWS credentials from AwsOptions.
- Returns:
An instance of Secret.
- class apache_beam.utils.secret.RawSecret(secret: str | bytes)[source]
Bases:
SecretSecret implementation wrapping a raw secret string or bytes directly.
- class apache_beam.utils.secret.GcpSecret(version_name: str)[source]
Bases:
SecretA secret manager implementation that retrieves secrets from Google Cloud Secret Manager.
Initializes a GcpSecret object.
- Parameters:
version_name – The full version name of the secret in Google Cloud Secret Manager. For example: projects/<id>/secrets/<secret_name>/versions/1. For more info, see https://cloud.google.com/python/docs/reference/secretmanager/latest/google.cloud.secretmanager_v1beta1.services.secret_manager_service.SecretManagerServiceClient#google_cloud_secretmanager_v1beta1_services_secret_manager_service_SecretManagerServiceClient_access_secret_version
- class apache_beam.utils.secret.AwsSecret(secret_id: str, version_id: str | None = None, version_stage: str | None = None, **client_settings: Any)[source]
Bases:
SecretA secret manager implementation that retrieves secrets from AWS Secrets Manager.
AWS credentials and connection settings (region, endpoint, SSL) come from the
--aws_*defaults inAwsOptionswhen pipeline options are given (seeapply_pipeline_options()), and otherwise from boto3’s defaults (e.g. environment variables, ~/.aws, or an IAM role). Connection settings can be overridden per secret with keyword arguments, e.g.AwsSecret('my-secret', region_name='us-west-2'), or in the secret spec, e.g.type:awssecret;name:my-secret;region_name:us-west-2.Initializes an AwsSecret object.
- Parameters:
secret_id – The ARN or name of the secret in AWS Secrets Manager.
version_id – The unique identifier of the version of the secret.
version_stage – The staging label of the version of the secret (e.g. ‘AWSCURRENT’).
**client_settings – AWS client connection settings that override the
--aws_*pipeline options for this secret. Names are AwsOptions setting names without theaws_prefix (e.g.region_name,endpoint_url,verify,disable_ssl). Credentials cannot be set here; use the--aws_*pipeline options instead.
- classmethod from_dict(spec_dict: dict[str, str]) AwsSecret[source]
Initialize AwsSecret from a dictionary specification.
Keys other than the secret identifiers are AWS client connection settings (see the
**client_settingskeyword arguments of__init__()).
- apply_pipeline_options(options: Any) None[source]
Uses AWS credentials and connection settings from AwsOptions.
Only the (unresolved) option values are stored; credential secret specs are resolved when the secret is fetched, e.g. on workers. Settings in
client_settingsstill take precedence.- Parameters:
options – PipelineOptions, or a dict of option names to values.
- class apache_beam.utils.secret.GcpHsmGeneratedSecret(project_id: str, location_id: str, key_ring_id: str, key_id: str, job_name: str)[source]
Bases:
SecretA secret manager implementation that generates a secret using a GCP HSM key and stores it in Google Cloud Secret Manager. If the secret already exists, it will be retrieved.
Initializes a GcpHsmGeneratedSecret object.
- Parameters:
project_id – The GCP project ID.
location_id – The GCP location ID for the HSM key.
key_ring_id – The ID of the KMS key ring.
key_id – The ID of the KMS key.
job_name – The name of the job, used to generate a unique secret name.
- classmethod from_dict(spec_dict: dict[str, str]) GcpHsmGeneratedSecret[source]
Initialize GcpHsmGeneratedSecret from a dictionary specification.
- get_secret_bytes() bytes[source]
Retrieves the secret bytes.
If the secret version already exists in Secret Manager, it is retrieved. Otherwise, a new secret and version are created. The new secret is generated using the HSM key.
- Returns:
The secret as a byte string.
- generate_dek(dek_size: int = 32) bytes[source]
Generates a new Data Encryption Key (DEK) using an HSM-backed key.
This function follows a key derivation process that incorporates entropy from the HSM-backed key into the nonce used for key derivation.
- Parameters:
dek_size – The size of the DEK to generate.
- Returns:
A new DEK of the specified size, url-safe base64-encoded.