org.apache.beam.sdk.io.aws2.options

Interface AwsOptions

All Superinterfaces:

HasDisplayData, PipelineOptions

All Known Subinterfaces:

S3Options

public interface AwsOptions
extends PipelineOptions

Options used to configure Amazon Web Services specific options such as credentials and region.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	AwsOptions.AwsRegionFactory Attempt to load default region.
static class	AwsOptions.AwsUserCredentialsFactory Return DefaultCredentialsProvider as default provider.

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Method Summary

Modifier and Type	Method and Description
AwsCredentialsProvider	getAwsCredentialsProvider() AwsCredentialsProvider used to configure AWS service clients.
Region	getAwsRegion() Region used to configure AWS service clients.
java.lang.Class<? extends ClientBuilderFactory>	getClientBuilderFactory()
java.net.URI	getEndpoint() Endpoint used to configure AWS service clients.
HttpClientConfiguration	getHttpClientConfiguration() HttpClientConfiguration used to configure AWS service clients.
ProxyConfiguration	getProxyConfiguration() ProxyConfiguration used to configure AWS service clients.
void	setAwsCredentialsProvider(AwsCredentialsProvider value)
void	setAwsRegion(Region region)
void	setClientBuilderFactory(java.lang.Class<? extends ClientBuilderFactory> clazz)
void	setEndpoint(java.net.URI uri)
void	setHttpClientConfiguration(HttpClientConfiguration value)
void	setProxyConfiguration(ProxyConfiguration value)

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Method Detail

getAwsRegion

@Default.InstanceFactory(value=AwsOptions.AwsRegionFactory.class)
Region getAwsRegion()

Region used to configure AWS service clients.

setAwsRegion

void setAwsRegion(Region region)

getEndpoint

java.net.URI getEndpoint()

Endpoint used to configure AWS service clients.

setEndpoint

void setEndpoint(java.net.URI uri)

getAwsCredentialsProvider

@Default.InstanceFactory(value=AwsOptions.AwsUserCredentialsFactory.class)
AwsCredentialsProvider getAwsCredentialsProvider()

AwsCredentialsProvider used to configure AWS service clients.

The class name of the provider must be set in the @type field. Note: Not all available providers are supported and some configuration options might be ignored.

Most providers must use the system environment following AWS conventions. Programmatic configuration for these providers is NOT supported:

DefaultCredentialsProvider EnvironmentVariableCredentialsProvider SystemPropertyCredentialsProvider ContainerCredentialsProvider

Example:

 --awsCredentialsProvider={"@type": "EnvironmentVariableCredentialsProvider"}
     

Some other providers support additional configuration:

StaticCredentialsProvider

Examples:

 --awsCredentialsProvider={
   "@type": "StaticCredentialsProvider",
   "awsAccessKeyId": "key_id_value",
   "awsSecretKey": "secret_value"
 }

 --awsCredentialsProvider={
   "@type": "StaticCredentialsProvider",
   "awsAccessKeyId": "key_id_value",
   "awsSecretKey": "secret_value",
   "sessionToken": "token_value"
 }

ProfileCredentialsProvider

profileName is optional, if not set the environment default is used. Be careful if using this provider programmatically, it can behave unexpectedly.

Examples:

 --awsCredentialsProvider={
   "@type": "ProfileCredentialsProvider"
 }

 --awsCredentialsProvider={
   "@type": "ProfileCredentialsProvider",
   "profileName": "my_profile"
 }

StsAssumeRoleCredentialsProvider

 --awsCredentialsProvider={
   "@type": "StsAssumeRoleCredentialsProvider",
   "roleArn": "role_arn_Value",
   "roleSessionName": "session_name_value",
   "policy": "policy_value",
   "durationSeconds": 3600
 }

StsAssumeRoleWithWebIdentityCredentialsProvider

Please note that this works for batch pipelines which can be completed within the expiration of the web identity token. Long batch or streaming pipelines wouldn't work with this Provider.

 --awsCredentialsProvider={
   "@type": "StsAssumeRoleWithWebIdentityCredentialsProvider",
   "roleArn": "role_arn_Value",
   "roleSessionName": "session_name_value",
   "webIdentityToken": "web_identity_token_value",
   "durationSeconds": 3600
 }

See Also:

DefaultCredentialsProvider

setAwsCredentialsProvider

void setAwsCredentialsProvider(AwsCredentialsProvider value)

getProxyConfiguration

ProxyConfiguration getProxyConfiguration()

ProxyConfiguration used to configure AWS service clients.

Note, only the options shown in the example below are supported. username and password are optional.

Example:

 --proxyConfiguration={
   "endpoint": "http://hostname:port",
   "username": "username",
   "password": "password"
 }

setProxyConfiguration

void setProxyConfiguration(ProxyConfiguration value)

getHttpClientConfiguration

HttpClientConfiguration getHttpClientConfiguration()

HttpClientConfiguration used to configure AWS service clients.

Example:

 --httpClientConfiguration={"socketTimeout":1000, "maxConnections":10}

setHttpClientConfiguration

void setHttpClientConfiguration(HttpClientConfiguration value)

getClientBuilderFactory

@Default.Class(value=ClientBuilderFactory.DefaultClientBuilder.class)
java.lang.Class<? extends ClientBuilderFactory> getClientBuilderFactory()

setClientBuilderFactory

void setClientBuilderFactory(java.lang.Class<? extends ClientBuilderFactory> clazz)