org.apache.beam.runners.dataflow.options

Interface DataflowWorkerHarnessOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineOptions, DataflowPipelineWorkerPoolOptions, DataflowProfilingOptions, DataflowWorkerLoggingOptions, ExperimentalOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions, PubsubOptions, StreamingOptions

@Hidden
public interface DataflowWorkerHarnessOptions
extends DataflowPipelineOptions

Options that are used exclusively within the Dataflow worker harness. These options have no effect at pipeline creation time.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineOptions

DataflowPipelineOptions.DefaultGcpRegionFactory, DataflowPipelineOptions.FlexResourceSchedulingGoal, DataflowPipelineOptions.StagingLocationFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.StagerFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

GcsOptions.ExecutorServiceFactory, GcsOptions.PathValidatorFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

DataflowProfilingOptions.DataflowProfilingAgentConfiguration

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

GcpOptions.DefaultProjectFactory, GcpOptions.EnableStreamingEngineFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Field Summary

Fields inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

STATE_CACHE_SIZE

Fields inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

STREAMING_ENGINE_EXPERIMENT, WINDMILL_SERVICE_EXPERIMENT

Method Summary

Modifier and Type	Method and Description
java.lang.String	getJobId() The identity of the Dataflow job.
java.lang.Integer	getWorkerCacheMb() The size of the worker's in-memory cache, in megabytes.
java.lang.String	getWorkerId() The identity of the worker running this pipeline.
void	setJobId(java.lang.String value)
void	setWorkerCacheMb(java.lang.Integer value)
void	setWorkerId(java.lang.String value)

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineOptions

getCreateFromSnapshot, getDataflowWorkerJar, getFlexRSGoal, getLabels, getPipelineUrl, getProject, getRegion, getServiceAccount, getStagingLocation, getTemplateLocation, isUpdate, setCreateFromSnapshot, setDataflowWorkerJar, setFlexRSGoal, setLabels, setPipelineUrl, setProject, setRegion, setServiceAccount, setStagingLocation, setTemplateLocation, setUpdate

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowClient, getDataflowEndpoint, getDataflowJobFile, getDumpHeapOnOOM, getNumberOfWorkerHarnessThreads, getOverrideWindmillBinary, getSaveHeapDumpsToGcsPath, getStager, getStagerClass, getTransformNameMapping, getWindmillServiceEndpoint, getWindmillServicePort, setApiRootUrl, setDataflowClient, setDataflowEndpoint, setDataflowJobFile, setDumpHeapOnOOM, setNumberOfWorkerHarnessThreads, setOverrideWindmillBinary, setSaveHeapDumpsToGcsPath, setStager, setStagerClass, setTransformNameMapping, setWindmillServiceEndpoint, setWindmillServicePort

Methods inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

addExperiment, getExperiments, getExperimentValue, hasExperiment, setExperiments

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getDiskSizeGb, getFilesToStage, getMaxNumWorkers, getMinCpuPlatform, getNetwork, getNumWorkers, getSubnetwork, getUsePublicIps, getWorkerDiskType, getWorkerHarnessContainerImage, getWorkerMachineType, setAutoscalingAlgorithm, setDiskSizeGb, setFilesToStage, setMaxNumWorkers, setMinCpuPlatform, setNetwork, setNumWorkers, setSubnetwork, setUsePublicIps, setWorkerDiskType, setWorkerHarnessContainerImage, setWorkerMachineType

Methods inherited from interface org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions

getHTTPWriteTimeout, getInsertBundleParallelism, getMaxStreamingBatchSize, getMaxStreamingRowsToBatch, getNumStreamingKeys, getTempDatasetId, setHTTPWriteTimeout, setInsertBundleParallelism, setMaxStreamingBatchSize, setMaxStreamingRowsToBatch, setNumStreamingKeys, setTempDatasetId

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

getExecutorService, getGcsEndpoint, getGcsPerformanceMetrics, getGcsUploadBufferSizeBytes, getGcsUtil, getPathValidator, getPathValidatorClass, getUseGrpcForGcs, setExecutorService, setGcsEndpoint, setGcsPerformanceMetrics, setGcsUploadBufferSizeBytes, setGcsUtil, setPathValidator, setPathValidatorClass, setUseGrpcForGcs

Methods inherited from interface org.apache.beam.runners.dataflow.options.CloudDebuggerOptions

getDebuggee, getEnableCloudDebugger, getMaxConditionCost, setDebuggee, setEnableCloudDebugger, setMaxConditionCost

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, getWorkerSystemErrMessageLevel, getWorkerSystemOutMessageLevel, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides, setWorkerSystemErrMessageLevel, setWorkerSystemOutMessageLevel

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

getProfilingAgentConfiguration, getSaveProfilesToGcs, setProfilingAgentConfiguration, setSaveProfilesToGcs

Methods inherited from interface org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions

getPubsubRootUrl, setPubsubRootUrl

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

getCredentialFactoryClass, getDataflowKmsKey, getGcpCredential, getGcpTempLocation, getWorkerRegion, getWorkerZone, getZone, isEnableStreamingEngine, setCredentialFactoryClass, setDataflowKmsKey, setEnableStreamingEngine, setGcpCredential, setGcpTempLocation, setWorkerRegion, setWorkerZone, setZone

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Method Detail

getWorkerId

java.lang.String getWorkerId()

The identity of the worker running this pipeline.

setWorkerId

void setWorkerId(java.lang.String value)

getJobId

java.lang.String getJobId()

The identity of the Dataflow job.

setJobId

void setJobId(java.lang.String value)

getWorkerCacheMb

@Default.Integer(value=100)
java.lang.Integer getWorkerCacheMb()

The size of the worker's in-memory cache, in megabytes.

Currently, this cache is used for storing read values of side inputs.

setWorkerCacheMb

void setWorkerCacheMb(java.lang.Integer value)