apache_beam.io.aws.clients.client_options module
Builds boto3 client arguments from AwsOptions.
Shared across AWS service clients (e.g. the S3 filesystem client) so that every boto3 client created by Beam uses the same credentials and connection settings.
- apache_beam.io.aws.clients.client_options.get_connection_setting_defaults() Mapping[str, Any][source]
Returns the non-credential client settings and their default values.
The names (e.g.
region_name,endpoint_url) come from AwsOptions.get_client_settings, so new settings added to AwsOptions are picked up automatically. The result is computed once and is read-only.
- apache_beam.io.aws.clients.client_options.parse_connection_settings(settings: Mapping[str, Any]) dict[str, Any][source]
Validates connection settings, e.g. from a secret spec.
Values for boolean settings (e.g.
disable_ssl) may be strings such as ‘true’ or ‘false’.- Parameters:
settings – A dict of client setting names (without the
aws_prefix) to values.- Returns:
A dict of the settings with values converted to the setting types.
- Raises:
ValueError – If a setting is a credential or not a known setting, or a boolean value is invalid.
- apache_beam.io.aws.clients.client_options.view_options(options: PipelineOptions | dict, options_cls: type[PipelineOptions]) PipelineOptions[source]
Returns an
options_clsview of PipelineOptions or a dict of values.
- apache_beam.io.aws.clients.client_options.get_client_kwargs(options: PipelineOptions | dict, service_options_cls: type[PipelineOptions] | None = None, overrides: dict[str, Any] | None = None) dict[str, Any][source]
Returns common boto3 client keyword arguments.
Settings come from the –aws_* defaults in AwsOptions, with the overrides from
service_options_clsapplied (e.g. –s3_* in S3Options; see AwsOptions.get_client_settings), and thenoverrides. Credentials are then resolved through –aws_credentials_secret_manager if it is set. Service-specific settings that are not shared (e.g. –s3_api_version) are not included.- Parameters:
options – PipelineOptions, or a dict of option names to values.
service_options_cls – Optional options class whose overrides apply, e.g. S3Options. If None, only the –aws_* defaults are used.
overrides – Optional dict of connection settings (setting names without the
aws_prefix, e.g.region_name) that take precedence over the options. Values that are None are ignored. Credentials cannot be overridden; see parse_connection_settings.