Package org.apache.beam.runners.dataflow.options

Interface DataflowWorkerHarnessOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, DataflowPipelineDebugOptions, DataflowPipelineOptions, DataflowPipelineWorkerPoolOptions, DataflowProfilingOptions, DataflowStreamingPipelineOptions, DataflowWorkerLoggingOptions, ExperimentalOptions, FileStagingOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, HasDisplayData, MemoryMonitorOptions, PipelineOptions, PubsubOptions, StreamingOptions

@Hidden
public interface DataflowWorkerHarnessOptions
extends DataflowPipelineOptions

Options that are used exclusively within the Dataflow worker harness. These options have no effect at pipeline creation time.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.StagerFactory, DataflowPipelineDebugOptions.UnboundedReaderMaxReadTimeFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineOptions

DataflowPipelineOptions.FlexResourceSchedulingGoal, DataflowPipelineOptions.StagingLocationFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

DataflowProfilingOptions.DataflowProfilingAgentConfiguration

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowStreamingPipelineOptions

DataflowStreamingPipelineOptions.EnableWindmillServiceDirectPathFactory, DataflowStreamingPipelineOptions.GlobalConfigRefreshPeriodFactory, DataflowStreamingPipelineOptions.HarnessUpdateReportingPeriodFactory, DataflowStreamingPipelineOptions.LocalWindmillHostportFactory, DataflowStreamingPipelineOptions.MaxStackTraceDepthToReportFactory, DataflowStreamingPipelineOptions.PeriodicStatusPageDirectoryFactory, DataflowStreamingPipelineOptions.WindmillServiceStreamingRpcBatchLimitFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

GcpOptions.DefaultProjectFactory, GcpOptions.EnableStreamingEngineFactory, GcpOptions.GcpOAuthScopesFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

GcsOptions.ExecutorServiceFactory, GcsOptions.GcsCustomAuditEntries, GcsOptions.PathValidatorFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

STATE_CACHE_SIZE, STATE_SAMPLING_PERIOD_MILLIS

Fields inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

STREAMING_ENGINE_EXPERIMENT, WINDMILL_SERVICE_EXPERIMENT

Method Summary

Modifier and Type	Method	Description
String	getJobId()	The identity of the Dataflow job.
String	getWorkerId()	The identity of the worker running this pipeline.
String	getWorkerPool()	The identity of the worker pool of this worker.
void	setJobId(String value)
void	setWorkerId(String value)
void	setWorkerPool(String value)

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions

getBigQueryEndpoint, getBigQueryProject, getBqStreamingApiLoggingFrequencySec, getEnableStorageReadApiV2, getGroupFilesFileLoad, getHTTPReadTimeout, getHTTPWriteTimeout, getInsertBundleParallelism, getJobLabelsMap, getMaxBufferingDurationMilliSec, getMaxConnectionPoolConnections, getMaxStreamingBatchSize, getMaxStreamingRowsToBatch, getMinConnectionPoolConnections, getNumStorageWriteApiStreamAppendClients, getNumStorageWriteApiStreams, getNumStreamingKeys, getStorageApiAppendThresholdBytes, getStorageApiAppendThresholdRecordCount, getStorageWriteApiMaxRequestSize, getStorageWriteApiMaxRetries, getStorageWriteApiTriggeringFrequencySec, getStorageWriteMaxInflightBytes, getStorageWriteMaxInflightRequests, getTempDatasetId, getUseStorageApiConnectionPool, getUseStorageWriteApi, getUseStorageWriteApiAtLeastOnce, setBigQueryEndpoint, setBigQueryProject, setBqStreamingApiLoggingFrequencySec, setEnableStorageReadApiV2, setGroupFilesFileLoad, setHTTPReadTimeout, setHTTPWriteTimeout, setInsertBundleParallelism, setJobLabelsMap, setMaxBufferingDurationMilliSec, setMaxConnectionPoolConnections, setMaxStreamingBatchSize, setMaxStreamingRowsToBatch, setMinConnectionPoolConnections, setNumStorageWriteApiStreamAppendClients, setNumStorageWriteApiStreams, setNumStreamingKeys, setStorageApiAppendThresholdBytes, setStorageApiAppendThresholdRecordCount, setStorageWriteApiMaxRequestSize, setStorageWriteApiMaxRetries, setStorageWriteApiTriggeringFrequencySec, setStorageWriteMaxInflightBytes, setStorageWriteMaxInflightRequests, setTempDatasetId, setUseStorageApiConnectionPool, setUseStorageWriteApi, setUseStorageWriteApiAtLeastOnce

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowClient, getDataflowJobFile, getDesiredNumUnboundedSourceSplits, getDumpHeapOnOOM, getJfrRecordingDurationSec, getNumberOfWorkerHarnessThreads, getReaderCacheTimeoutSec, getRecordJfrOnGcThrashing, getSaveHeapDumpsToGcsPath, getSdkHarnessContainerImageOverrides, getStager, getStagerClass, getTransformNameMapping, getUnboundedReaderMaxElements, getUnboundedReaderMaxReadTimeMs, getUnboundedReaderMaxReadTimeSec, getUnboundedReaderMaxWaitForElementsMs, getWorkerCacheMb, setApiRootUrl, setDataflowClient, setDataflowJobFile, setDesiredNumUnboundedSourceSplits, setDumpHeapOnOOM, setJfrRecordingDurationSec, setNumberOfWorkerHarnessThreads, setReaderCacheTimeoutSec, setRecordJfrOnGcThrashing, setSaveHeapDumpsToGcsPath, setSdkHarnessContainerImageOverrides, setStager, setStagerClass, setTransformNameMapping, setUnboundedReaderMaxElements, setUnboundedReaderMaxReadTimeMs, setUnboundedReaderMaxReadTimeSec, setUnboundedReaderMaxWaitForElementsMs, setWorkerCacheMb

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineOptions

getCreateFromSnapshot, getDataflowEndpoint, getDataflowServiceOptions, getDataflowWorkerJar, getFlexRSGoal, getJdkAddOpenModules, getLabels, getPipelineUrl, getProject, getRegion, getServiceAccount, getStagingLocation, getTemplateLocation, isHotKeyLoggingEnabled, isUpdate, setCreateFromSnapshot, setDataflowEndpoint, setDataflowServiceOptions, setDataflowWorkerJar, setFlexRSGoal, setHotKeyLoggingEnabled, setJdkAddOpenModules, setLabels, setPipelineUrl, setProject, setRegion, setServiceAccount, setStagingLocation, setTemplateLocation, setUpdate

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getDiskSizeGb, getMaxNumWorkers, getMinCpuPlatform, getNetwork, getNumWorkers, getSdkContainerImage, getSubnetwork, getUsePublicIps, getWorkerDiskType, getWorkerHarnessContainerImage, getWorkerMachineType, setAutoscalingAlgorithm, setDiskSizeGb, setMaxNumWorkers, setMinCpuPlatform, setNetwork, setNumWorkers, setSdkContainerImage, setSubnetwork, setUsePublicIps, setWorkerDiskType, setWorkerHarnessContainerImage, setWorkerMachineType

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

getProfilingAgentConfiguration, getSaveProfilesToGcs, setProfilingAgentConfiguration, setSaveProfilesToGcs

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowStreamingPipelineOptions

getActiveWorkRefreshPeriodMillis, getChannelzShowOnlyWindmillServiceChannels, getGlobalConfigRefreshPeriod, getIsWindmillServiceDirectPathEnabled, getLocalWindmillHostport, getMaxBundlesFromWindmillOutstanding, getMaxBytesFromWindmillOutstanding, getMaxStackTraceDepthToReport, getOverrideWindmillBinary, getPeriodicStatusPageOutputDirectory, getPerWorkerMetricsUpdateReportingPeriodMillis, getStreamingSideInputCacheExpirationMillis, getStreamingSideInputCacheMb, getStuckCommitDurationMillis, getUseSeparateWindmillHeartbeatStreams, getUseWindmillIsolatedChannels, getWindmillGetDataStreamCount, getWindmillHarnessUpdateReportingPeriod, getWindmillMessagesBetweenIsReadyChecks, getWindmillRequestBatchedGetWorkResponse, getWindmillServiceCommitThreads, getWindmillServiceEndpoint, getWindmillServicePort, getWindmillServiceRpcChannelAliveTimeoutSec, getWindmillServiceStreamingLogEveryNStreamFailures, getWindmillServiceStreamingRpcBatchLimit, getWindmillServiceStreamingRpcHealthCheckPeriodMs, getWindmillServiceStreamMaxBackoffMillis, setActiveWorkRefreshPeriodMillis, setChannelzShowOnlyWindmillServiceChannels, setGlobalConfigRefreshPeriod, setIsWindmillServiceDirectPathEnabled, setLocalWindmillHostport, setMaxBundlesFromWindmillOutstanding, setMaxBytesFromWindmillOutstanding, setMaxStackTraceDepthToReport, setOverrideWindmillBinary, setPeriodicStatusPageOutputDirectory, setPerWorkerMetricsUpdateReportingPeriodMillis, setStreamingSideInputCacheExpirationMillis, setStreamingSideInputCacheMb, setStuckCommitDurationMillis, setUseSeparateWindmillHeartbeatStreams, setUseWindmillIsolatedChannels, setWindmillGetDataStreamCount, setWindmillHarnessUpdateReportingPeriod, setWindmillMessagesBetweenIsReadyChecks, setWindmillRequestBatchedGetWorkResponse, setWindmillServiceCommitThreads, setWindmillServiceEndpoint, setWindmillServicePort, setWindmillServiceRpcChannelAliveTimeoutSec, setWindmillServiceStreamingLogEveryNStreamFailures, setWindmillServiceStreamingRpcBatchLimit, setWindmillServiceStreamingRpcHealthCheckPeriodMs, setWindmillServiceStreamMaxBackoffMillis

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, getWorkerSystemErrMessageLevel, getWorkerSystemOutMessageLevel, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides, setWorkerSystemErrMessageLevel, setWorkerSystemOutMessageLevel

Methods inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

getExperiments, setExperiments

Methods inherited from interface org.apache.beam.sdk.options.FileStagingOptions

getFilesToStage, setFilesToStage

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

getCredentialFactoryClass, getDataflowKmsKey, getGcpCredential, getGcpOauthScopes, getGcpTempLocation, getImpersonateServiceAccount, getWorkerRegion, getWorkerZone, getZone, isEnableStreamingEngine, setCredentialFactoryClass, setDataflowKmsKey, setEnableStreamingEngine, setGcpCredential, setGcpOauthScopes, setGcpTempLocation, setImpersonateServiceAccount, setWorkerRegion, setWorkerZone, setZone

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

getEnableBucketReadMetricCounter, getEnableBucketWriteMetricCounter, getExecutorService, getGcsCustomAuditEntries, getGcsEndpoint, getGcsHttpRequestReadTimeout, getGcsHttpRequestWriteTimeout, getGcsPerformanceMetrics, getGcsReadCounterPrefix, getGcsRewriteDataOpBatchLimit, getGcsUploadBufferSizeBytes, getGcsUtil, getGcsWriteCounterPrefix, getGoogleCloudStorageReadOptions, getPathValidator, getPathValidatorClass, setEnableBucketReadMetricCounter, setEnableBucketWriteMetricCounter, setExecutorService, setGcsCustomAuditEntries, setGcsEndpoint, setGcsHttpRequestReadTimeout, setGcsHttpRequestWriteTimeout, setGcsPerformanceMetrics, setGcsReadCounterPrefix, setGcsRewriteDataOpBatchLimit, setGcsUploadBufferSizeBytes, setGcsUtil, setGcsWriteCounterPrefix, setGoogleCloudStorageReadOptions, setPathValidator, setPathValidatorClass

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface org.apache.beam.sdk.options.MemoryMonitorOptions

getGCThrashingPercentagePerPeriod, getGzipCompressHeapDumps, getRemoteHeapDumpLocation, setGCThrashingPercentagePerPeriod, setGzipCompressHeapDumps, setRemoteHeapDumpLocation

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions

getPubsubRootUrl, setPubsubRootUrl

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

getUpdateCompatibilityVersion, isStreaming, setStreaming, setUpdateCompatibilityVersion

Method Details

getWorkerId

String getWorkerId()

The identity of the worker running this pipeline.

setWorkerId

void setWorkerId(String value)

getJobId

String getJobId()

The identity of the Dataflow job.

setJobId

void setJobId(String value)

getWorkerPool

String getWorkerPool()

The identity of the worker pool of this worker.

setWorkerPool

void setWorkerPool(String value)