Package org.apache.beam.sdk.io.aws2.options

Interface AwsOptions

All Superinterfaces:
HasDisplayData, PipelineOptions
All Known Subinterfaces:
S3Options
public interface AwsOptions extends PipelineOptions
Options used to configure Amazon Web Services specific options such as credentials and region.

  • Method Details

    • getAwsRegion

      @InstanceFactory(AwsRegionFactory.class) Region getAwsRegion()
      Region used to configure AWS service clients.

    • setAwsRegion

      void setAwsRegion(Region region)

    • getEndpoint

      URI getEndpoint()
      Endpoint used to configure AWS service clients.

    • setEndpoint

      void setEndpoint(URI uri)

    • getAwsCredentialsProvider

      @InstanceFactory(AwsUserCredentialsFactory.class) AwsCredentialsProvider getAwsCredentialsProvider()
      AwsCredentialsProvider used to configure AWS service clients.

      The class name of the provider must be set in the @type field. Note: Not all available providers are supported and some configuration options might be ignored.

      Most providers must use the system environment following AWS conventions. Programmatic configuration for these providers is NOT supported:

    • DefaultCredentialsProvider
    • EnvironmentVariableCredentialsProvider
    • SystemPropertyCredentialsProvider
    • ContainerCredentialsProvider

      Example: 

      --awsCredentialsProvider={"@type": "EnvironmentVariableCredentialsProvider"}

      Some other providers support additional configuration:

    • StaticCredentialsProvider

      Examples: 

      --awsCredentialsProvider={
   "@type": "StaticCredentialsProvider",
   "accessKeyId": "key_id_value",
   "secretAccessKey": "secret_value"
 }

 --awsCredentialsProvider={
   "@type": "StaticCredentialsProvider",
   "accessKeyId": "key_id_value",
   "secretAccessKey": "secret_value",
   "sessionToken": "token_value"
 }
    • ProfileCredentialsProvider

      profileName is optional, if not set the environment default is used. Be careful if using this provider programmatically, it can behave unexpectedly.

      Examples: 

      --awsCredentialsProvider={
   "@type": "ProfileCredentialsProvider"
 }

 --awsCredentialsProvider={
   "@type": "ProfileCredentialsProvider",
   "profileName": "my_profile"
 }
    • StsAssumeRoleCredentialsProvider 
      --awsCredentialsProvider={
   "@type": "StsAssumeRoleCredentialsProvider",
   "roleArn": "role_arn_Value",
   "roleSessionName": "session_name_value",
   "policy": "policy_value",
   "durationSeconds": 3600
 }
    • StsAssumeRoleWithWebIdentityCredentialsProvider

      Please note that this works for batch pipelines which can be completed within the expiration of the web identity token. Long batch or streaming pipelines wouldn't work with this Provider. 

      --awsCredentialsProvider={
   "@type": "StsAssumeRoleWithWebIdentityCredentialsProvider",
   "roleArn": "role_arn_Value",
   "roleSessionName": "session_name_value",
   "webIdentityToken": "web_identity_token_value",
   "durationSeconds": 3600
 }
    • StsAssumeRoleForFederatedCredentialsProvider

      Similar to StsAssumeRoleWithWebIdentityCredentialsProvider but supporting long-running jobs. The configuration expects the FQCN for an implementation of WebIdTokenProvider which will be used to retrieve a fresh token with each STS auth request. 

      --awsCredentialsProvider={
   "@type": "StsAssumeRoleWithDynamicWebIdentityCredentialsProvider",
   "roleArn": "role_arn_Value",
   "audience": "audience_value",
   "webIdTokenProviderFQCN": "fully_qualified_class_name",
   "durationSeconds": 3600
 }
      • See Also:

    • setAwsCredentialsProvider

      void setAwsCredentialsProvider(AwsCredentialsProvider value)

    • getProxyConfiguration

      ProxyConfiguration getProxyConfiguration()
      ProxyConfiguration used to configure AWS service clients.

      Note, only the options shown in the example below are supported. username and password are optional.

      Example: 

      --proxyConfiguration={
   "endpoint": "http://hostname:port",
   "username": "username",
   "password": "password"
 }

    • setProxyConfiguration

      void setProxyConfiguration(ProxyConfiguration value)

    • getHttpClientConfiguration

      HttpClientConfiguration getHttpClientConfiguration()
      HttpClientConfiguration used to configure AWS service clients.

      Example: 

      --httpClientConfiguration={"socketTimeout":1000, "maxConnections":10}

    • setHttpClientConfiguration

      void setHttpClientConfiguration(HttpClientConfiguration value)

    • getClientBuilderFactory

      @Class(DefaultClientBuilder.class) Class<? extends ClientBuilderFactory> getClientBuilderFactory()

    • setClientBuilderFactory

      void setClientBuilderFactory(Class<? extends ClientBuilderFactory> clazz)