org.apache.beam.runners.dataflow.options

Interface DataflowPipelineOptions

All Superinterfaces:

ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, DataflowProfilingOptions, DataflowWorkerLoggingOptions, ExperimentalOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions, PubsubOptions, StreamingOptions

All Known Subinterfaces:

DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineOptions
extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, CloudDebuggerOptions, DataflowWorkerLoggingOptions, DataflowProfilingOptions, PubsubOptions

Options that can be used to configure the DataflowRunner.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineOptions.StagingLocationFactory Returns a default staging location under GcpOptions.getGcpTempLocation().

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.StagerFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

GcsOptions.ExecutorServiceFactory, GcsOptions.PathValidatorFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides

Nested classes/interfaces inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

DataflowProfilingOptions.DataflowProfilingAgentConfiguration

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

GcpOptions.DefaultProjectFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Method Summary

Modifier and Type	Method and Description
java.util.Map<java.lang.String,java.lang.String>	getLabels() Labels that will be applied to the billing records for this job.
java.lang.String	getPipelineUrl() The URL of the staged portable pipeline.
java.lang.String	getProject() Project id to use when launching jobs.
java.lang.String	getRegion() The Google Compute Engine region for creating Dataflow jobs.
java.lang.String	getServiceAccount() Run the job as a specific service account, instead of the default GCE robot.
java.lang.String	getStagingLocation() GCS path for staging local files, e.g.
java.lang.String	getTemplateLocation() Where the runner should generate a template file.
boolean	isUpdate() Whether to update the currently running pipeline with the same name as this one.
void	setLabels(java.util.Map<java.lang.String,java.lang.String> labels)
void	setPipelineUrl(java.lang.String urlString)
void	setProject(java.lang.String value)
void	setRegion(java.lang.String region)
void	setServiceAccount(java.lang.String value)
void	setStagingLocation(java.lang.String value)
void	setTemplateLocation(java.lang.String value)
void	setUpdate(boolean value)

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions

getApiRootUrl, getDataflowClient, getDataflowEndpoint, getDataflowJobFile, getDumpHeapOnOOM, getNumberOfWorkerHarnessThreads, getOverrideWindmillBinary, getSaveHeapDumpsToGcsPath, getStager, getStagerClass, getTransformNameMapping, getWindmillServiceEndpoint, getWindmillServicePort, setApiRootUrl, setDataflowClient, setDataflowEndpoint, setDataflowJobFile, setDumpHeapOnOOM, setNumberOfWorkerHarnessThreads, setOverrideWindmillBinary, setSaveHeapDumpsToGcsPath, setStager, setStagerClass, setTransformNameMapping, setWindmillServiceEndpoint, setWindmillServicePort

Methods inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

getExperiments, hasExperiment, setExperiments

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions

getAutoscalingAlgorithm, getDiskSizeGb, getFilesToStage, getMaxNumWorkers, getNetwork, getNumWorkers, getSubnetwork, getUsePublicIps, getWorkerDiskType, getWorkerHarnessContainerImage, getWorkerMachineType, getZone, setAutoscalingAlgorithm, setDiskSizeGb, setFilesToStage, setMaxNumWorkers, setNetwork, setNumWorkers, setSubnetwork, setUsePublicIps, setWorkerDiskType, setWorkerHarnessContainerImage, setWorkerMachineType, setZone

Methods inherited from interface org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions

getTempDatasetId, setTempDatasetId

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcsOptions

getExecutorService, getGcsEndpoint, getGcsUploadBufferSizeBytes, getGcsUtil, getPathValidator, getPathValidatorClass, setExecutorService, setGcsEndpoint, setGcsUploadBufferSizeBytes, setGcsUtil, setPathValidator, setPathValidatorClass

Methods inherited from interface org.apache.beam.runners.dataflow.options.CloudDebuggerOptions

getDebuggee, getEnableCloudDebugger, getMaxConditionCost, setDebuggee, setEnableCloudDebugger, setMaxConditionCost

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions

getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, getWorkerSystemErrMessageLevel, getWorkerSystemOutMessageLevel, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides, setWorkerSystemErrMessageLevel, setWorkerSystemOutMessageLevel

Methods inherited from interface org.apache.beam.runners.dataflow.options.DataflowProfilingOptions

getProfilingAgentConfiguration, getSaveProfilesToGcs, setProfilingAgentConfiguration, setSaveProfilesToGcs

Methods inherited from interface org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions

getPubsubRootUrl, setPubsubRootUrl

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

getCredentialFactoryClass, getGcpCredential, getGcpTempLocation, getZone, setCredentialFactoryClass, setGcpCredential, setGcpTempLocation, setZone

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Method Detail

getProject

@Validation.Required
 @Default.InstanceFactory(value=GcpOptions.DefaultProjectFactory.class)
java.lang.String getProject()

Description copied from interface: GcpOptions

Project id to use when launching jobs.

Specified by:

getProject in interface GcpOptions

setProject

void setProject(java.lang.String value)

Specified by:

setProject in interface GcpOptions

getStagingLocation

@Default.InstanceFactory(value=DataflowPipelineOptions.StagingLocationFactory.class)
java.lang.String getStagingLocation()

GCS path for staging local files, e.g. gs://bucket/object

Must be a valid Cloud Storage URL, beginning with the prefix "gs://"

If getStagingLocation() is not set, it will default to GcpOptions.getGcpTempLocation(). GcpOptions.getGcpTempLocation() must be a valid GCS path.

setStagingLocation

void setStagingLocation(java.lang.String value)

isUpdate

boolean isUpdate()

Whether to update the currently running pipeline with the same name as this one.

setUpdate

void setUpdate(boolean value)

getTemplateLocation

java.lang.String getTemplateLocation()

Where the runner should generate a template file. Must either be local or Cloud Storage.

setTemplateLocation

void setTemplateLocation(java.lang.String value)

getServiceAccount

@Hidden
 @Experimental
java.lang.String getServiceAccount()

Run the job as a specific service account, instead of the default GCE robot.

setServiceAccount

void setServiceAccount(java.lang.String value)

getRegion

@Hidden
 @Experimental
 @Default.String(value="us-central1")
java.lang.String getRegion()

The Google Compute Engine region for creating Dataflow jobs.

NOTE: The Cloud Dataflow now also supports the region flag.

setRegion

void setRegion(java.lang.String region)

getLabels

java.util.Map<java.lang.String,java.lang.String> getLabels()

Labels that will be applied to the billing records for this job.

setLabels

void setLabels(java.util.Map<java.lang.String,java.lang.String> labels)

getPipelineUrl

java.lang.String getPipelineUrl()

The URL of the staged portable pipeline.

setPipelineUrl

void setPipelineUrl(java.lang.String urlString)