Package org.apache.beam.sdk.extensions.gcp.options

Interface GcsOptions

All Superinterfaces:

ApplicationNameOptions, GcpOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions

All Known Subinterfaces:

DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface GcsOptions
extends ApplicationNameOptions, GcpOptions, PipelineOptions

Options used to configure Google Cloud Storage.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	GcsOptions.ExecutorServiceFactory	Returns the default ExecutorService to use within the Apache Beam SDK.
static class	GcsOptions.GcsCustomAuditEntries	Creates a GcsOptions.GcsCustomAuditEntries that key-value pairs to be stored as custom information in GCS audit logs.
static class	GcsOptions.PathValidatorFactory	Creates a PathValidator object using the class specified in getPathValidatorClass().

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

GcpOptions.DefaultProjectFactory, GcpOptions.EnableStreamingEngineFactory, GcpOptions.GcpOAuthScopesFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

STREAMING_ENGINE_EXPERIMENT, WINDMILL_SERVICE_EXPERIMENT

Method Summary

Modifier and Type	Method	Description
Boolean	getEnableBucketReadMetricCounter()	If true, reports number of bytes read from each gcs bucket.
Boolean	getEnableBucketWriteMetricCounter()	If true, reports number of bytes written to each gcs bucket.
ExecutorService	getExecutorService()	Deprecated. use ExecutorOptions.getScheduledExecutorService() instead
GcsOptions.GcsCustomAuditEntries	getGcsCustomAuditEntries()
String	getGcsEndpoint()	GCS endpoint to use.
@Nullable Integer	getGcsHttpRequestReadTimeout()
@Nullable Integer	getGcsHttpRequestWriteTimeout()
Boolean	getGcsPerformanceMetrics()	If true, reports metrics of certain operations, such as batch copies.
String	getGcsReadCounterPrefix()
@Nullable Integer	getGcsRewriteDataOpBatchLimit()
@Nullable Integer	getGcsUploadBufferSizeBytes()	The buffer size (in bytes) to use when uploading files to GCS.
GcsUtil	getGcsUtil()	The GcsUtil instance that should be used to communicate with Google Cloud Storage.
String	getGcsWriteCounterPrefix()
GoogleCloudStorageReadOptions	getGoogleCloudStorageReadOptions()
PathValidator	getPathValidator()	The path validator instance that should be used to validate paths.
Class<? extends PathValidator>	getPathValidatorClass()	The class of the validator that should be created and used to validate paths.
void	setEnableBucketReadMetricCounter(Boolean enableBucketReadMetricCounter)
void	setEnableBucketWriteMetricCounter(Boolean enableBucketWriteMetricCounter)
void	setExecutorService(ExecutorService value)	Deprecated. use ExecutorOptions.setScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) instead.
void	setGcsCustomAuditEntries(GcsOptions.GcsCustomAuditEntries entries)
void	setGcsEndpoint(String value)
void	setGcsHttpRequestReadTimeout(@Nullable Integer timeoutMs)
void	setGcsHttpRequestWriteTimeout(@Nullable Integer timeoutMs)
void	setGcsPerformanceMetrics(Boolean reportPerformanceMetrics)
void	setGcsReadCounterPrefix(String gcsReadCounterPrefix)
void	setGcsRewriteDataOpBatchLimit(@Nullable Integer timeoutMs)
void	setGcsUploadBufferSizeBytes(@Nullable Integer bytes)
void	setGcsUtil(GcsUtil value)
void	setGcsWriteCounterPrefix(String gcsReadCounterPrefix)
void	setGoogleCloudStorageReadOptions(GoogleCloudStorageReadOptions value)
void	setPathValidator(PathValidator validator)
void	setPathValidatorClass(Class<? extends PathValidator> validatorClass)

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

getCredentialFactoryClass, getDataflowKmsKey, getGcpCredential, getGcpOauthScopes, getGcpTempLocation, getImpersonateServiceAccount, getProject, getWorkerRegion, getWorkerZone, getZone, isEnableStreamingEngine, setCredentialFactoryClass, setDataflowKmsKey, setEnableStreamingEngine, setGcpCredential, setGcpOauthScopes, setGcpTempLocation, setImpersonateServiceAccount, setProject, setWorkerRegion, setWorkerZone, setZone

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getGbek, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setGbek, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Method Details

getGcsUtil

@InstanceFactory(GcsUtilFactory.class)
@Hidden
GcsUtil getGcsUtil()

The GcsUtil instance that should be used to communicate with Google Cloud Storage.

setGcsUtil

void setGcsUtil(GcsUtil value)

getGoogleCloudStorageReadOptions

@InstanceFactory(GcsReadOptionsFactory.class)
@Hidden
GoogleCloudStorageReadOptions getGoogleCloudStorageReadOptions()

setGoogleCloudStorageReadOptions

void setGoogleCloudStorageReadOptions(GoogleCloudStorageReadOptions value)

getExecutorService

@InstanceFactory(ExecutorServiceFactory.class)
@Hidden
@Deprecated
ExecutorService getExecutorService()

Deprecated.

use ExecutorOptions.getScheduledExecutorService() instead

The ExecutorService instance to use to create threads, can be overridden to specify an ExecutorService that is compatible with the user's environment. If unset, the default is to use ExecutorOptions.getScheduledExecutorService().

setExecutorService

@Deprecated
void setExecutorService(ExecutorService value)

Deprecated.

use ExecutorOptions.setScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) instead. If set, it may result in multiple ExecutorServices, and therefore thread pools, in the runtime.

getGcsEndpoint

@Hidden
String getGcsEndpoint()

GCS endpoint to use. If unspecified, uses the default endpoint.

setGcsEndpoint

void setGcsEndpoint(String value)

getGcsUploadBufferSizeBytes

@Nullable Integer getGcsUploadBufferSizeBytes()

The buffer size (in bytes) to use when uploading files to GCS. Please see the documentation for AsyncWriteChannelOptions.getUploadChunkSize() for more information on the restrictions and performance implications of this value.

setGcsUploadBufferSizeBytes

void setGcsUploadBufferSizeBytes(@Nullable Integer bytes)

getPathValidatorClass

@Class(GcsPathValidator.class)
Class<? extends PathValidator> getPathValidatorClass()

The class of the validator that should be created and used to validate paths. If pathValidator has not been set explicitly, an instance of this class will be constructed and used as the path validator.

setPathValidatorClass

void setPathValidatorClass(Class<? extends PathValidator> validatorClass)

getPathValidator

@InstanceFactory(PathValidatorFactory.class)
PathValidator getPathValidator()

The path validator instance that should be used to validate paths. If no path validator has been set explicitly, the default is to use the instance factory that constructs a path validator based upon the currently set pathValidatorClass.

setPathValidator

void setPathValidator(PathValidator validator)

getGcsPerformanceMetrics

@Boolean(false)
Boolean getGcsPerformanceMetrics()

If true, reports metrics of certain operations, such as batch copies.

setGcsPerformanceMetrics

void setGcsPerformanceMetrics(Boolean reportPerformanceMetrics)

getGcsHttpRequestReadTimeout

@Nullable Integer getGcsHttpRequestReadTimeout()

setGcsHttpRequestReadTimeout

void setGcsHttpRequestReadTimeout(@Nullable Integer timeoutMs)

getGcsHttpRequestWriteTimeout

@Nullable Integer getGcsHttpRequestWriteTimeout()

setGcsHttpRequestWriteTimeout

void setGcsHttpRequestWriteTimeout(@Nullable Integer timeoutMs)

getGcsRewriteDataOpBatchLimit

@Nullable Integer getGcsRewriteDataOpBatchLimit()

setGcsRewriteDataOpBatchLimit

void setGcsRewriteDataOpBatchLimit(@Nullable Integer timeoutMs)

getEnableBucketWriteMetricCounter

@Boolean(false)
Boolean getEnableBucketWriteMetricCounter()

If true, reports number of bytes written to each gcs bucket.

setEnableBucketWriteMetricCounter

void setEnableBucketWriteMetricCounter(Boolean enableBucketWriteMetricCounter)

getEnableBucketReadMetricCounter

@Boolean(false)
Boolean getEnableBucketReadMetricCounter()

If true, reports number of bytes read from each gcs bucket.

setEnableBucketReadMetricCounter

void setEnableBucketReadMetricCounter(Boolean enableBucketReadMetricCounter)

getGcsReadCounterPrefix

@String("GCS_read_bytes_counter")
String getGcsReadCounterPrefix()

setGcsReadCounterPrefix

void setGcsReadCounterPrefix(String gcsReadCounterPrefix)

getGcsWriteCounterPrefix

@String("GCS_write_bytes_counter")
String getGcsWriteCounterPrefix()

setGcsWriteCounterPrefix

void setGcsWriteCounterPrefix(String gcsReadCounterPrefix)

getGcsCustomAuditEntries

GcsOptions.GcsCustomAuditEntries getGcsCustomAuditEntries()

setGcsCustomAuditEntries

void setGcsCustomAuditEntries(GcsOptions.GcsCustomAuditEntries entries)