org.apache.beam.sdk.extensions.gcp.options

Interface GcsOptions

All Superinterfaces:

ApplicationNameOptions, GcpOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions

All Known Subinterfaces:

DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface GcsOptions
extends ApplicationNameOptions, GcpOptions, PipelineOptions

Options used to configure Google Cloud Storage.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	GcsOptions.ExecutorServiceFactory Returns the default ExecutorService to use within the Apache Beam SDK.
static class	GcsOptions.PathValidatorFactory Creates a PathValidator object using the class specified in getPathValidatorClass().

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

GcpOptions.DefaultProjectFactory, GcpOptions.EnableStreamingEngineFactory, GcpOptions.GcpOAuthScopesFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

STREAMING_ENGINE_EXPERIMENT, WINDMILL_SERVICE_EXPERIMENT

Method Summary

Modifier and Type	Method and Description
java.lang.Boolean	getEnableBucketReadMetricCounter() If true, reports number of bytes read from each gcs bucket.
java.lang.Boolean	getEnableBucketWriteMetricCounter() If true, reports number of bytes written to each gcs bucket.
java.util.concurrent.ExecutorService	getExecutorService() Deprecated. use ExecutorOptions.getScheduledExecutorService() instead
java.lang.String	getGcsEndpoint() GCS endpoint to use.
@Nullable java.lang.Integer	getGcsHttpRequestReadTimeout()
@Nullable java.lang.Integer	getGcsHttpRequestWriteTimeout()
java.lang.Boolean	getGcsPerformanceMetrics() If true, reports metrics of certain operations, such as batch copies.
java.lang.String	getGcsReadCounterPrefix()
@Nullable java.lang.Integer	getGcsRewriteDataOpBatchLimit()
@Nullable java.lang.Integer	getGcsUploadBufferSizeBytes() The buffer size (in bytes) to use when uploading files to GCS.
GcsUtil	getGcsUtil() The GcsUtil instance that should be used to communicate with Google Cloud Storage.
java.lang.String	getGcsWriteCounterPrefix()
PathValidator	getPathValidator() The path validator instance that should be used to validate paths.
java.lang.Class<? extends PathValidator>	getPathValidatorClass() The class of the validator that should be created and used to validate paths.
void	setEnableBucketReadMetricCounter(java.lang.Boolean enableBucketReadMetricCounter)
void	setEnableBucketWriteMetricCounter(java.lang.Boolean enableBucketWriteMetricCounter)
void	setExecutorService(java.util.concurrent.ExecutorService value) Deprecated. use ExecutorOptions.setScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) instead. If set, it may result in multiple ExecutorServices, and therefore thread pools, in the runtime.
void	setGcsEndpoint(java.lang.String value)
void	setGcsHttpRequestReadTimeout(@Nullable java.lang.Integer timeoutMs)
void	setGcsHttpRequestWriteTimeout(@Nullable java.lang.Integer timeoutMs)
void	setGcsPerformanceMetrics(java.lang.Boolean reportPerformanceMetrics)
void	setGcsReadCounterPrefix(java.lang.String gcsReadCounterPrefix)
void	setGcsRewriteDataOpBatchLimit(@Nullable java.lang.Integer timeoutMs)
void	setGcsUploadBufferSizeBytes(@Nullable java.lang.Integer bytes)
void	setGcsUtil(GcsUtil value)
void	setGcsWriteCounterPrefix(java.lang.String gcsReadCounterPrefix)
void	setPathValidator(PathValidator validator)
void	setPathValidatorClass(java.lang.Class<? extends PathValidator> validatorClass)

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

getCredentialFactoryClass, getDataflowKmsKey, getGcpCredential, getGcpOauthScopes, getGcpTempLocation, getImpersonateServiceAccount, getProject, getWorkerRegion, getWorkerZone, getZone, isEnableStreamingEngine, setCredentialFactoryClass, setDataflowKmsKey, setEnableStreamingEngine, setGcpCredential, setGcpOauthScopes, setGcpTempLocation, setImpersonateServiceAccount, setProject, setWorkerRegion, setWorkerZone, setZone

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Method Detail

getGcsUtil

@Default.InstanceFactory(value=GcsUtil.GcsUtilFactory.class)
 @Hidden
GcsUtil getGcsUtil()

The GcsUtil instance that should be used to communicate with Google Cloud Storage.

setGcsUtil

void setGcsUtil(GcsUtil value)

getExecutorService

@Default.InstanceFactory(value=GcsOptions.ExecutorServiceFactory.class)
 @Hidden
 @Deprecated
java.util.concurrent.ExecutorService getExecutorService()

Deprecated. use ExecutorOptions.getScheduledExecutorService() instead

The ExecutorService instance to use to create threads, can be overridden to specify an ExecutorService that is compatible with the user's environment. If unset, the default is to use ExecutorOptions.getScheduledExecutorService().

setExecutorService

@Deprecated
void setExecutorService(java.util.concurrent.ExecutorService value)

Deprecated. use ExecutorOptions.setScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) instead. If set, it may result in multiple ExecutorServices, and therefore thread pools, in the runtime.

getGcsEndpoint

@Hidden
java.lang.String getGcsEndpoint()

GCS endpoint to use. If unspecified, uses the default endpoint.

setGcsEndpoint

void setGcsEndpoint(java.lang.String value)

getGcsUploadBufferSizeBytes

@Nullable java.lang.Integer getGcsUploadBufferSizeBytes()

The buffer size (in bytes) to use when uploading files to GCS. Please see the documentation for AsyncWriteChannelOptions.getUploadChunkSize() for more information on the restrictions and performance implications of this value.

setGcsUploadBufferSizeBytes

void setGcsUploadBufferSizeBytes(@Nullable java.lang.Integer bytes)

getPathValidatorClass

@Default.Class(value=GcsPathValidator.class)
java.lang.Class<? extends PathValidator> getPathValidatorClass()

The class of the validator that should be created and used to validate paths. If pathValidator has not been set explicitly, an instance of this class will be constructed and used as the path validator.

setPathValidatorClass

void setPathValidatorClass(java.lang.Class<? extends PathValidator> validatorClass)

getPathValidator

@Default.InstanceFactory(value=GcsOptions.PathValidatorFactory.class)
PathValidator getPathValidator()

The path validator instance that should be used to validate paths. If no path validator has been set explicitly, the default is to use the instance factory that constructs a path validator based upon the currently set pathValidatorClass.

setPathValidator

void setPathValidator(PathValidator validator)

getGcsPerformanceMetrics

@Default.Boolean(value=false)
java.lang.Boolean getGcsPerformanceMetrics()

If true, reports metrics of certain operations, such as batch copies.

setGcsPerformanceMetrics

void setGcsPerformanceMetrics(java.lang.Boolean reportPerformanceMetrics)

getGcsHttpRequestReadTimeout

@Nullable java.lang.Integer getGcsHttpRequestReadTimeout()

setGcsHttpRequestReadTimeout

void setGcsHttpRequestReadTimeout(@Nullable java.lang.Integer timeoutMs)

getGcsHttpRequestWriteTimeout

@Nullable java.lang.Integer getGcsHttpRequestWriteTimeout()

setGcsHttpRequestWriteTimeout

void setGcsHttpRequestWriteTimeout(@Nullable java.lang.Integer timeoutMs)

getGcsRewriteDataOpBatchLimit

@Nullable java.lang.Integer getGcsRewriteDataOpBatchLimit()

setGcsRewriteDataOpBatchLimit

void setGcsRewriteDataOpBatchLimit(@Nullable java.lang.Integer timeoutMs)

getEnableBucketWriteMetricCounter

@Default.Boolean(value=false)
java.lang.Boolean getEnableBucketWriteMetricCounter()

If true, reports number of bytes written to each gcs bucket.

setEnableBucketWriteMetricCounter

void setEnableBucketWriteMetricCounter(java.lang.Boolean enableBucketWriteMetricCounter)

getEnableBucketReadMetricCounter

@Default.Boolean(value=false)
java.lang.Boolean getEnableBucketReadMetricCounter()

If true, reports number of bytes read from each gcs bucket.

setEnableBucketReadMetricCounter

void setEnableBucketReadMetricCounter(java.lang.Boolean enableBucketReadMetricCounter)

getGcsReadCounterPrefix

@Default.String(value="GCS_read_bytes_counter")
java.lang.String getGcsReadCounterPrefix()

setGcsReadCounterPrefix

void setGcsReadCounterPrefix(java.lang.String gcsReadCounterPrefix)

getGcsWriteCounterPrefix

@Default.String(value="GCS_write_bytes_counter")
java.lang.String getGcsWriteCounterPrefix()

setGcsWriteCounterPrefix

void setGcsWriteCounterPrefix(java.lang.String gcsReadCounterPrefix)