org.apache.beam.runners.dataflow.options

Interface DataflowPipelineOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, DataflowProfilingOptions, DataflowWorkerLoggingOptions, ExperimentalOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions, PubsubOptions, StreamingOptions

All Known Subinterfaces:

DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineOptions
extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, CloudDebuggerOptions, DataflowWorkerLoggingOptions, DataflowProfilingOptions, PubsubOptions

Options that can be used to configure the DataflowRunner.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineOptions.FlexResourceSchedulingGoal Set of available Flexible Resource Scheduling goals.
static class	DataflowPipelineOptions.StagingLocationFactory Returns a default staging location under GcpOptions.getGcpTempLocation().

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.StagerFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

GcsOptions.ExecutorServiceFactory, GcsOptions.PathValidatorFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

DataflowProfilingOptions.DataflowProfilingAgentConfiguration

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

GcpOptions.DefaultProjectFactory, GcpOptions.EnableStreamingEngineFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Field Summary

Fields inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

STATE_CACHE_SIZE

Fields inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

STREAMING_ENGINE_EXPERIMENT, WINDMILL_SERVICE_EXPERIMENT

Method Summary

Modifier and Type	Method and Description
java.lang.String	getCreateFromSnapshot() If set, the snapshot from which the job should be created.
java.lang.String	getDataflowWorkerJar()
DataflowPipelineOptions.FlexResourceSchedulingGoal	getFlexRSGoal() This option controls Flexible Resource Scheduling mode.
java.util.Map<java.lang.String,java.lang.String>	getLabels() Labels that will be applied to the billing records for this job.
java.lang.String	getPipelineUrl() The URL of the staged portable pipeline.
java.lang.String	getProject() Project id to use when launching jobs.
java.lang.String	getRegion() The Google Compute Engine region for creating Dataflow jobs.
java.lang.String	getServiceAccount() Run the job as a specific service account, instead of the default GCE robot.
java.lang.String	getStagingLocation() GCS path for staging local files, e.g.
java.lang.String	getTemplateLocation() Where the runner should generate a template file.
boolean	isHotKeyLoggingEnabled() If enabled then the literal key will be logged to Cloud Logging if a hot key is detected.
boolean	isUpdate() Whether to update the currently running pipeline with the same name as this one.
void	setCreateFromSnapshot(java.lang.String value)
void	setDataflowWorkerJar(java.lang.String dataflowWorkerJar)
void	setFlexRSGoal(DataflowPipelineOptions.FlexResourceSchedulingGoal goal)
void	setHotKeyLoggingEnabled(boolean value)
void	setLabels(java.util.Map<java.lang.String,java.lang.String> labels)
void	setPipelineUrl(java.lang.String urlString)
void	setProject(java.lang.String value)
void	setRegion(java.lang.String region)
void	setServiceAccount(java.lang.String value)
void	setStagingLocation(java.lang.String value)
void	setTemplateLocation(java.lang.String value)
void	setUpdate(boolean value)

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowClient, getDataflowEndpoint, getDataflowJobFile, getDumpHeapOnOOM, getGCThrashingPercentagePerPeriod, getNumberOfWorkerHarnessThreads, getOverrideWindmillBinary, getReaderCacheTimeoutSec, getSaveHeapDumpsToGcsPath, getStager, getStagerClass, getTransformNameMapping, getWindmillServiceEndpoint, getWindmillServicePort, getWorkerCacheMb, setApiRootUrl, setDataflowClient, setDataflowEndpoint, setDataflowJobFile, setDumpHeapOnOOM, setGCThrashingPercentagePerPeriod, setNumberOfWorkerHarnessThreads, setOverrideWindmillBinary, setReaderCacheTimeoutSec, setSaveHeapDumpsToGcsPath, setStager, setStagerClass, setTransformNameMapping, setWindmillServiceEndpoint, setWindmillServicePort, setWorkerCacheMb

Methods inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

addExperiment, getExperiments, getExperimentValue, hasExperiment, setExperiments

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getDiskSizeGb, getFilesToStage, getMaxNumWorkers, getMinCpuPlatform, getNetwork, getNumWorkers, getSubnetwork, getUsePublicIps, getWorkerDiskType, getWorkerHarnessContainerImage, getWorkerMachineType, setAutoscalingAlgorithm, setDiskSizeGb, setFilesToStage, setMaxNumWorkers, setMinCpuPlatform, setNetwork, setNumWorkers, setSubnetwork, setUsePublicIps, setWorkerDiskType, setWorkerHarnessContainerImage, setWorkerMachineType

Methods inherited from interface org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions

getHTTPWriteTimeout, getInsertBundleParallelism, getLatencyLoggingFrequency, getMaxStreamingBatchSize, getMaxStreamingRowsToBatch, getNumStreamingKeys, getTempDatasetId, setHTTPWriteTimeout, setInsertBundleParallelism, setLatencyLoggingFrequency, setMaxStreamingBatchSize, setMaxStreamingRowsToBatch, setNumStreamingKeys, setTempDatasetId

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

getExecutorService, getGcsEndpoint, getGcsPerformanceMetrics, getGcsUploadBufferSizeBytes, getGcsUtil, getPathValidator, getPathValidatorClass, setExecutorService, setGcsEndpoint, setGcsPerformanceMetrics, setGcsUploadBufferSizeBytes, setGcsUtil, setPathValidator, setPathValidatorClass

Methods inherited from interface org.apache.beam.runners.dataflow.options.CloudDebuggerOptions

getDebuggee, getEnableCloudDebugger, getMaxConditionCost, setDebuggee, setEnableCloudDebugger, setMaxConditionCost

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, getWorkerSystemErrMessageLevel, getWorkerSystemOutMessageLevel, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides, setWorkerSystemErrMessageLevel, setWorkerSystemOutMessageLevel

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

getProfilingAgentConfiguration, getSaveProfilesToGcs, setProfilingAgentConfiguration, setSaveProfilesToGcs

Methods inherited from interface org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions

getPubsubRootUrl, setPubsubRootUrl, targetForRootUrl

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

getCredentialFactoryClass, getDataflowKmsKey, getGcpCredential, getGcpTempLocation, getWorkerRegion, getWorkerZone, getZone, isEnableStreamingEngine, setCredentialFactoryClass, setDataflowKmsKey, setEnableStreamingEngine, setGcpCredential, setGcpTempLocation, setWorkerRegion, setWorkerZone, setZone

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Method Detail

getProject

@Validation.Required
 @Default.InstanceFactory(value=GcpOptions.DefaultProjectFactory.class)
java.lang.String getProject()

Description copied from interface: GcpOptions

Project id to use when launching jobs.

Specified by:

getProject in interface GcpOptions

setProject

void setProject(java.lang.String value)

Specified by:

setProject in interface GcpOptions

getStagingLocation

@Default.InstanceFactory(value=DataflowPipelineOptions.StagingLocationFactory.class)
java.lang.String getStagingLocation()

GCS path for staging local files, e.g. gs://bucket/object

Must be a valid Cloud Storage URL, beginning with the prefix "gs://"

If getStagingLocation() is not set, it will default to GcpOptions.getGcpTempLocation(). GcpOptions.getGcpTempLocation() must be a valid GCS path.

setStagingLocation

void setStagingLocation(java.lang.String value)

isUpdate

boolean isUpdate()

Whether to update the currently running pipeline with the same name as this one.

setUpdate

void setUpdate(boolean value)

getCreateFromSnapshot

@Hidden
 @Experimental
java.lang.String getCreateFromSnapshot()

If set, the snapshot from which the job should be created.

setCreateFromSnapshot

void setCreateFromSnapshot(java.lang.String value)

getTemplateLocation

java.lang.String getTemplateLocation()

Where the runner should generate a template file. Must either be local or Cloud Storage.

setTemplateLocation

void setTemplateLocation(java.lang.String value)

getServiceAccount

@Hidden
 @Experimental
java.lang.String getServiceAccount()

Run the job as a specific service account, instead of the default GCE robot.

setServiceAccount

void setServiceAccount(java.lang.String value)

getRegion

@Default.InstanceFactory(value=DefaultGcpRegionFactory.class)
java.lang.String getRegion()

The Google Compute Engine region for creating Dataflow jobs.

setRegion

void setRegion(java.lang.String region)

getLabels

java.util.Map<java.lang.String,java.lang.String> getLabels()

Labels that will be applied to the billing records for this job.

setLabels

void setLabels(java.util.Map<java.lang.String,java.lang.String> labels)

getPipelineUrl

java.lang.String getPipelineUrl()

The URL of the staged portable pipeline.

setPipelineUrl

void setPipelineUrl(java.lang.String urlString)

getDataflowWorkerJar

java.lang.String getDataflowWorkerJar()

setDataflowWorkerJar

void setDataflowWorkerJar(java.lang.String dataflowWorkerJar)

getFlexRSGoal

@Default.Enum(value="UNSPECIFIED")
DataflowPipelineOptions.FlexResourceSchedulingGoal getFlexRSGoal()

This option controls Flexible Resource Scheduling mode.

setFlexRSGoal

void setFlexRSGoal(DataflowPipelineOptions.FlexResourceSchedulingGoal goal)

isHotKeyLoggingEnabled

boolean isHotKeyLoggingEnabled()

If enabled then the literal key will be logged to Cloud Logging if a hot key is detected.

setHotKeyLoggingEnabled

void setHotKeyLoggingEnabled(boolean value)