org.apache.beam.runners.dataflow.options

Interface DataflowPipelineOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, DataflowProfilingOptions, DataflowStreamingPipelineOptions, DataflowWorkerLoggingOptions, ExperimentalOptions, FileStagingOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, HasDisplayData, MemoryMonitorOptions, PipelineOptions, PubsubOptions, StreamingOptions

All Known Subinterfaces:

DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineOptions
extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, DataflowWorkerLoggingOptions, DataflowStreamingPipelineOptions, DataflowProfilingOptions, PubsubOptions

Options that can be used to configure the DataflowRunner.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineOptions.FlexResourceSchedulingGoal Set of available Flexible Resource Scheduling goals.
static class	DataflowPipelineOptions.StagingLocationFactory Returns a default staging location under GcpOptions.getGcpTempLocation().

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.StagerFactory, DataflowPipelineDebugOptions.UnboundedReaderMaxReadTimeFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

GcsOptions.ExecutorServiceFactory, GcsOptions.GcsCustomAuditEntries, GcsOptions.PathValidatorFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowStreamingPipelineOptions

DataflowStreamingPipelineOptions.EnableWindmillServiceDirectPathFactory, DataflowStreamingPipelineOptions.GlobalConfigRefreshPeriodFactory, DataflowStreamingPipelineOptions.HarnessUpdateReportingPeriodFactory, DataflowStreamingPipelineOptions.LocalWindmillHostportFactory, DataflowStreamingPipelineOptions.MaxStackTraceDepthToReportFactory, DataflowStreamingPipelineOptions.PeriodicStatusPageDirectoryFactory, DataflowStreamingPipelineOptions.WindmillServiceStreamingRpcBatchLimitFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

DataflowProfilingOptions.DataflowProfilingAgentConfiguration

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

GcpOptions.DefaultProjectFactory, GcpOptions.EnableStreamingEngineFactory, GcpOptions.GcpOAuthScopesFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Field Summary

Fields inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

STATE_CACHE_SIZE, STATE_SAMPLING_PERIOD_MILLIS

Fields inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

STREAMING_ENGINE_EXPERIMENT, WINDMILL_SERVICE_EXPERIMENT

Method Summary

Modifier and Type	Method and Description
java.lang.String	getCreateFromSnapshot() If set, the snapshot from which the job should be created.
java.lang.String	getDataflowEndpoint() Dataflow endpoint to use.
java.util.List<java.lang.String>	getDataflowServiceOptions() Service options are set by the user and configure the service.
java.lang.String	getDataflowWorkerJar()
DataflowPipelineOptions.FlexResourceSchedulingGoal	getFlexRSGoal() This option controls Flexible Resource Scheduling mode.
java.util.List<java.lang.String>	getJdkAddOpenModules() Open modules needed for reflection that access JDK internals with Java 9+
java.util.Map<java.lang.String,java.lang.String>	getLabels() Labels that will be applied to the billing records for this job.
java.lang.String	getPipelineUrl() The URL of the staged portable pipeline.
java.lang.String	getProject() Project id to use when launching jobs.
java.lang.String	getRegion() The Google Compute Engine region for creating Dataflow jobs.
java.lang.String	getServiceAccount() Run the job as a specific service account, instead of the default GCE robot.
java.lang.String	getStagingLocation() GCS path for staging local files, e.g.
java.lang.String	getTemplateLocation() Where the runner should generate a template file.
boolean	isHotKeyLoggingEnabled() If enabled then the literal key will be logged to Cloud Logging if a hot key is detected.
boolean	isUpdate() Whether to update the currently running pipeline with the same name as this one.
void	setCreateFromSnapshot(java.lang.String value)
void	setDataflowEndpoint(java.lang.String value)
void	setDataflowServiceOptions(java.util.List<java.lang.String> options)
void	setDataflowWorkerJar(java.lang.String dataflowWorkerJar)
void	setFlexRSGoal(DataflowPipelineOptions.FlexResourceSchedulingGoal goal)
void	setHotKeyLoggingEnabled(boolean value)
void	setJdkAddOpenModules(java.util.List<java.lang.String> options)
void	setLabels(java.util.Map<java.lang.String,java.lang.String> labels)
void	setPipelineUrl(java.lang.String urlString)
void	setProject(java.lang.String value)
void	setRegion(java.lang.String region)
void	setServiceAccount(java.lang.String value)
void	setStagingLocation(java.lang.String value)
void	setTemplateLocation(java.lang.String value) Sets the Cloud Storage path where the Dataflow template will be stored.
void	setUpdate(boolean value)

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowClient, getDataflowJobFile, getDesiredNumUnboundedSourceSplits, getDumpHeapOnOOM, getJfrRecordingDurationSec, getNumberOfWorkerHarnessThreads, getReaderCacheTimeoutSec, getRecordJfrOnGcThrashing, getSaveHeapDumpsToGcsPath, getSdkHarnessContainerImageOverrides, getStager, getStagerClass, getTransformNameMapping, getUnboundedReaderMaxElements, getUnboundedReaderMaxReadTimeMs, getUnboundedReaderMaxReadTimeSec, getUnboundedReaderMaxWaitForElementsMs, getWorkerCacheMb, setApiRootUrl, setDataflowClient, setDataflowJobFile, setDesiredNumUnboundedSourceSplits, setDumpHeapOnOOM, setJfrRecordingDurationSec, setNumberOfWorkerHarnessThreads, setReaderCacheTimeoutSec, setRecordJfrOnGcThrashing, setSaveHeapDumpsToGcsPath, setSdkHarnessContainerImageOverrides, setStager, setStagerClass, setTransformNameMapping, setUnboundedReaderMaxElements, setUnboundedReaderMaxReadTimeMs, setUnboundedReaderMaxReadTimeSec, setUnboundedReaderMaxWaitForElementsMs, setWorkerCacheMb

Methods inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

addExperiment, getExperiments, getExperimentValue, hasExperiment, setExperiments

Methods inherited from interface org.apache.beam.sdk.options.MemoryMonitorOptions

getGCThrashingPercentagePerPeriod, setGCThrashingPercentagePerPeriod

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getDiskSizeGb, getMaxNumWorkers, getMinCpuPlatform, getNetwork, getNumWorkers, getSdkContainerImage, getSubnetwork, getUsePublicIps, getWorkerDiskType, getWorkerHarnessContainerImage, getWorkerMachineType, setAutoscalingAlgorithm, setDiskSizeGb, setMaxNumWorkers, setMinCpuPlatform, setNetwork, setNumWorkers, setSdkContainerImage, setSubnetwork, setUsePublicIps, setWorkerDiskType, setWorkerHarnessContainerImage, setWorkerMachineType

Methods inherited from interface org.apache.beam.sdk.options.FileStagingOptions

getFilesToStage, setFilesToStage

Methods inherited from interface org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions

getBigQueryEndpoint, getBigQueryProject, getBqStreamingApiLoggingFrequencySec, getEnableStorageReadApiV2, getGroupFilesFileLoad, getHTTPReadTimeout, getHTTPWriteTimeout, getInsertBundleParallelism, getJobLabelsMap, getMaxBufferingDurationMilliSec, getMaxConnectionPoolConnections, getMaxStreamingBatchSize, getMaxStreamingRowsToBatch, getMinConnectionPoolConnections, getNumStorageWriteApiStreamAppendClients, getNumStorageWriteApiStreams, getNumStreamingKeys, getStorageApiAppendThresholdBytes, getStorageApiAppendThresholdRecordCount, getStorageWriteApiMaxRequestSize, getStorageWriteApiMaxRetries, getStorageWriteApiTriggeringFrequencySec, getStorageWriteMaxInflightBytes, getStorageWriteMaxInflightRequests, getTempDatasetId, getUseStorageApiConnectionPool, getUseStorageWriteApi, getUseStorageWriteApiAtLeastOnce, setBigQueryEndpoint, setBigQueryProject, setBqStreamingApiLoggingFrequencySec, setEnableStorageReadApiV2, setGroupFilesFileLoad, setHTTPReadTimeout, setHTTPWriteTimeout, setInsertBundleParallelism, setJobLabelsMap, setMaxBufferingDurationMilliSec, setMaxConnectionPoolConnections, setMaxStreamingBatchSize, setMaxStreamingRowsToBatch, setMinConnectionPoolConnections, setNumStorageWriteApiStreamAppendClients, setNumStorageWriteApiStreams, setNumStreamingKeys, setStorageApiAppendThresholdBytes, setStorageApiAppendThresholdRecordCount, setStorageWriteApiMaxRequestSize, setStorageWriteApiMaxRetries, setStorageWriteApiTriggeringFrequencySec, setStorageWriteMaxInflightBytes, setStorageWriteMaxInflightRequests, setTempDatasetId, setUseStorageApiConnectionPool, setUseStorageWriteApi, setUseStorageWriteApiAtLeastOnce

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

getEnableBucketReadMetricCounter, getEnableBucketWriteMetricCounter, getExecutorService, getGcsCustomAuditEntries, getGcsEndpoint, getGcsHttpRequestReadTimeout, getGcsHttpRequestWriteTimeout, getGcsPerformanceMetrics, getGcsReadCounterPrefix, getGcsRewriteDataOpBatchLimit, getGcsUploadBufferSizeBytes, getGcsUtil, getGcsWriteCounterPrefix, getGoogleCloudStorageReadOptions, getPathValidator, getPathValidatorClass, setEnableBucketReadMetricCounter, setEnableBucketWriteMetricCounter, setExecutorService, setGcsCustomAuditEntries, setGcsEndpoint, setGcsHttpRequestReadTimeout, setGcsHttpRequestWriteTimeout, setGcsPerformanceMetrics, setGcsReadCounterPrefix, setGcsRewriteDataOpBatchLimit, setGcsUploadBufferSizeBytes, setGcsUtil, setGcsWriteCounterPrefix, setGoogleCloudStorageReadOptions, setPathValidator, setPathValidatorClass

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, getWorkerSystemErrMessageLevel, getWorkerSystemOutMessageLevel, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides, setWorkerSystemErrMessageLevel, setWorkerSystemOutMessageLevel

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowStreamingPipelineOptions

getActiveWorkRefreshPeriodMillis, getChannelzShowOnlyWindmillServiceChannels, getGlobalConfigRefreshPeriod, getIsWindmillServiceDirectPathEnabled, getLocalWindmillHostport, getMaxBundlesFromWindmillOutstanding, getMaxBytesFromWindmillOutstanding, getMaxStackTraceDepthToReport, getOverrideWindmillBinary, getPeriodicStatusPageOutputDirectory, getPerWorkerMetricsUpdateReportingPeriodMillis, getStreamingSideInputCacheExpirationMillis, getStreamingSideInputCacheMb, getStuckCommitDurationMillis, getUseSeparateWindmillHeartbeatStreams, getUseWindmillIsolatedChannels, getWindmillGetDataStreamCount, getWindmillHarnessUpdateReportingPeriod, getWindmillMessagesBetweenIsReadyChecks, getWindmillRequestBatchedGetWorkResponse, getWindmillServiceCommitThreads, getWindmillServiceEndpoint, getWindmillServicePort, getWindmillServiceRpcChannelAliveTimeoutSec, getWindmillServiceStreamingLogEveryNStreamFailures, getWindmillServiceStreamingRpcBatchLimit, getWindmillServiceStreamingRpcHealthCheckPeriodMs, getWindmillServiceStreamMaxBackoffMillis, setActiveWorkRefreshPeriodMillis, setChannelzShowOnlyWindmillServiceChannels, setGlobalConfigRefreshPeriod, setIsWindmillServiceDirectPathEnabled, setLocalWindmillHostport, setMaxBundlesFromWindmillOutstanding, setMaxBytesFromWindmillOutstanding, setMaxStackTraceDepthToReport, setOverrideWindmillBinary, setPeriodicStatusPageOutputDirectory, setPerWorkerMetricsUpdateReportingPeriodMillis, setStreamingSideInputCacheExpirationMillis, setStreamingSideInputCacheMb, setStuckCommitDurationMillis, setUseSeparateWindmillHeartbeatStreams, setUseWindmillIsolatedChannels, setWindmillGetDataStreamCount, setWindmillHarnessUpdateReportingPeriod, setWindmillMessagesBetweenIsReadyChecks, setWindmillRequestBatchedGetWorkResponse, setWindmillServiceCommitThreads, setWindmillServiceEndpoint, setWindmillServicePort, setWindmillServiceRpcChannelAliveTimeoutSec, setWindmillServiceStreamingLogEveryNStreamFailures, setWindmillServiceStreamingRpcBatchLimit, setWindmillServiceStreamingRpcHealthCheckPeriodMs, setWindmillServiceStreamMaxBackoffMillis

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

getProfilingAgentConfiguration, getSaveProfilesToGcs, setProfilingAgentConfiguration, setSaveProfilesToGcs

Methods inherited from interface org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions

getPubsubRootUrl, setPubsubRootUrl, targetForRootUrl

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

getCredentialFactoryClass, getDataflowKmsKey, getGcpCredential, getGcpOauthScopes, getGcpTempLocation, getImpersonateServiceAccount, getWorkerRegion, getWorkerZone, getZone, isEnableStreamingEngine, setCredentialFactoryClass, setDataflowKmsKey, setEnableStreamingEngine, setGcpCredential, setGcpOauthScopes, setGcpTempLocation, setImpersonateServiceAccount, setWorkerRegion, setWorkerZone, setZone

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Method Detail

getProject

@Validation.Required
 @Default.InstanceFactory(value=GcpOptions.DefaultProjectFactory.class)
java.lang.String getProject()

Description copied from interface: GcpOptions

Project id to use when launching jobs.

Specified by:

getProject in interface GcpOptions

setProject

void setProject(java.lang.String value)

Specified by:

setProject in interface GcpOptions

getStagingLocation

@Default.InstanceFactory(value=DataflowPipelineOptions.StagingLocationFactory.class)
java.lang.String getStagingLocation()

GCS path for staging local files, e.g. gs://bucket/object

Must be a valid Cloud Storage URL, beginning with the prefix "gs://"

If getStagingLocation() is not set, it will default to GcpOptions.getGcpTempLocation(). GcpOptions.getGcpTempLocation() must be a valid GCS path.

setStagingLocation

void setStagingLocation(java.lang.String value)

isUpdate

boolean isUpdate()

Whether to update the currently running pipeline with the same name as this one.

setUpdate

void setUpdate(boolean value)

getCreateFromSnapshot

java.lang.String getCreateFromSnapshot()

If set, the snapshot from which the job should be created.

setCreateFromSnapshot

void setCreateFromSnapshot(java.lang.String value)

getTemplateLocation

java.lang.String getTemplateLocation()

Where the runner should generate a template file. Must either be local or Cloud Storage.

setTemplateLocation

void setTemplateLocation(java.lang.String value)

Sets the Cloud Storage path where the Dataflow template will be stored. Required for creating Flex Templates or Classic Templates.

Example:

 DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
 options.setTemplateLocation("gs://your-bucket/templates/my-template");
 

Parameters:

value - Cloud Storage path for storing the Dataflow template.

getDataflowServiceOptions

java.util.List<java.lang.String> getDataflowServiceOptions()

Service options are set by the user and configure the service. This decouples service side feature availability from the Apache Beam release cycle.

setDataflowServiceOptions

void setDataflowServiceOptions(java.util.List<java.lang.String> options)

getServiceAccount

java.lang.String getServiceAccount()

Run the job as a specific service account, instead of the default GCE robot.

setServiceAccount

void setServiceAccount(java.lang.String value)

getRegion

@Default.InstanceFactory(value=DefaultGcpRegionFactory.class)
java.lang.String getRegion()

The Google Compute Engine region for creating Dataflow jobs.

setRegion

void setRegion(java.lang.String region)

getDataflowEndpoint

@Default.String(value="")
java.lang.String getDataflowEndpoint()

Dataflow endpoint to use.

Defaults to the current version of the Google Cloud Dataflow API, at the time the current SDK version was released.

If the string contains "://", then this is treated as a URL, otherwise DataflowPipelineDebugOptions.getApiRootUrl() is used as the root URL.

Specified by:

getDataflowEndpoint in interface DataflowPipelineDebugOptions

setDataflowEndpoint

void setDataflowEndpoint(java.lang.String value)

Specified by:

setDataflowEndpoint in interface DataflowPipelineDebugOptions

getLabels

java.util.Map<java.lang.String,java.lang.String> getLabels()

Labels that will be applied to the billing records for this job.

setLabels

void setLabels(java.util.Map<java.lang.String,java.lang.String> labels)

getPipelineUrl

java.lang.String getPipelineUrl()

The URL of the staged portable pipeline.

setPipelineUrl

void setPipelineUrl(java.lang.String urlString)

getDataflowWorkerJar

java.lang.String getDataflowWorkerJar()

setDataflowWorkerJar

void setDataflowWorkerJar(java.lang.String dataflowWorkerJar)

getFlexRSGoal

@Default.Enum(value="UNSPECIFIED")
DataflowPipelineOptions.FlexResourceSchedulingGoal getFlexRSGoal()

This option controls Flexible Resource Scheduling mode.

setFlexRSGoal

void setFlexRSGoal(DataflowPipelineOptions.FlexResourceSchedulingGoal goal)

isHotKeyLoggingEnabled

boolean isHotKeyLoggingEnabled()

If enabled then the literal key will be logged to Cloud Logging if a hot key is detected.

setHotKeyLoggingEnabled

void setHotKeyLoggingEnabled(boolean value)

getJdkAddOpenModules

java.util.List<java.lang.String> getJdkAddOpenModules()

Open modules needed for reflection that access JDK internals with Java 9+

With JDK 16+, JDK internals are strongly encapsulated and can result in an InaccessibleObjectException being thrown if a tool or library uses reflection that access JDK internals. If you see these errors in your worker logs, you can pass in modules to open using the format module/package=target-module(,target-module)* to allow access to the library. E.g. java.base/java.lang=jamm

You may see warnings that jamm, a library used to more accurately size objects, is unable to make a private field accessible. To resolve the warning, open the specified module/package to jamm.

setJdkAddOpenModules

void setJdkAddOpenModules(java.util.List<java.lang.String> options)