org.apache.beam.runners.dataflow.options

Interface DataflowPipelineWorkerPoolOptions

All Superinterfaces:

GcpOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions

All Known Subinterfaces:

DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

public interface DataflowPipelineWorkerPoolOptions
extends GcpOptions

Options that are used to configure the Dataflow pipeline worker pool.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType Type of autoscaling algorithm to use.
static class	DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory Returns the default Docker container image that executes Dataflow worker harness, residing in Google Container Registry.

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

GcpOptions.DefaultProjectFactory, GcpOptions.EnableStreamingEngineFactory, GcpOptions.GcpTempLocationFactory, GcpOptions.GcpUserCredentialsFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

GoogleApiDebugOptions.GoogleApiTracer

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

STREAMING_ENGINE_EXPERIMENT, WINDMILL_SERVICE_EXPERIMENT

Method Summary

Modifier and Type	Method and Description
DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType	getAutoscalingAlgorithm() [Experimental] The autoscaling algorithm to use for the workerpool.
int	getDiskSizeGb() Remote worker disk size, in gigabytes, or 0 to use the default size.
java.util.List<java.lang.String>	getFilesToStage() List of local files to make available to workers.
int	getMaxNumWorkers() The maximum number of workers to use for the workerpool.
@Nullable java.lang.String	getMinCpuPlatform() Specifies a Minimum CPU platform for VM instances.
java.lang.String	getNetwork() GCE network for launching workers.
int	getNumWorkers() Number of workers to use when executing the Dataflow job.
java.lang.String	getSubnetwork() GCE subnetwork for launching workers.
@Nullable java.lang.Boolean	getUsePublicIps() Specifies whether worker pools should be started with public IP addresses.
java.lang.String	getWorkerDiskType() Specifies what type of persistent disk is used.
java.lang.String	getWorkerHarnessContainerImage() Docker container image that executes Dataflow worker harness, residing in Google Container Registry.
java.lang.String	getWorkerMachineType() Machine type to create Dataflow worker VMs as.
void	setAutoscalingAlgorithm(DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType value)
void	setDiskSizeGb(int value)
void	setFilesToStage(java.util.List<java.lang.String> value)
void	setMaxNumWorkers(int value)
void	setMinCpuPlatform(java.lang.String minCpuPlatform)
void	setNetwork(java.lang.String value)
void	setNumWorkers(int value)
void	setSubnetwork(java.lang.String value)
void	setUsePublicIps(@Nullable java.lang.Boolean value)
void	setWorkerDiskType(java.lang.String value)
void	setWorkerHarnessContainerImage(java.lang.String value)
void	setWorkerMachineType(java.lang.String value)

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

getCredentialFactoryClass, getDataflowKmsKey, getGcpCredential, getGcpTempLocation, getProject, getWorkerRegion, getWorkerZone, getZone, isEnableStreamingEngine, setCredentialFactoryClass, setDataflowKmsKey, setEnableStreamingEngine, setGcpCredential, setGcpTempLocation, setProject, setWorkerRegion, setWorkerZone, setZone

Methods inherited from interface org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions

getGoogleApiTrace, setGoogleApiTrace

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Method Detail

getNumWorkers

int getNumWorkers()

Number of workers to use when executing the Dataflow job. Note that selection of an autoscaling algorithm other then NONE will affect the size of the worker pool. If left unspecified, the Dataflow service will determine the number of workers.

setNumWorkers

void setNumWorkers(int value)

getAutoscalingAlgorithm

@Experimental(value=AUTOSCALING)
DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType getAutoscalingAlgorithm()

[Experimental] The autoscaling algorithm to use for the workerpool.

• NONE: does not change the size of the worker pool.
• BASIC: autoscale the worker pool size up to maxNumWorkers until the job completes.
• THROUGHPUT_BASED: autoscale the workerpool based on throughput (up to maxNumWorkers).

setAutoscalingAlgorithm

void setAutoscalingAlgorithm(DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType value)

getMaxNumWorkers

int getMaxNumWorkers()

The maximum number of workers to use for the workerpool. This options limits the size of the workerpool for the lifetime of the job, including pipeline updates. If left unspecified, the Dataflow service will compute a ceiling.

setMaxNumWorkers

void setMaxNumWorkers(int value)

getDiskSizeGb

int getDiskSizeGb()

Remote worker disk size, in gigabytes, or 0 to use the default size.

setDiskSizeGb

void setDiskSizeGb(int value)

getWorkerHarnessContainerImage

@Default.InstanceFactory(value=DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory.class)
 @Hidden
java.lang.String getWorkerHarnessContainerImage()

Docker container image that executes Dataflow worker harness, residing in Google Container Registry.

setWorkerHarnessContainerImage

void setWorkerHarnessContainerImage(java.lang.String value)

getNetwork

java.lang.String getNetwork()

GCE network for launching workers.

Default is up to the Dataflow service.

setNetwork

void setNetwork(java.lang.String value)

getSubnetwork

java.lang.String getSubnetwork()

GCE subnetwork for launching workers.

Default is up to the Dataflow service. Expected format is regions/REGION/subnetworks/SUBNETWORK or the fully qualified subnetwork name, beginning with https://..., e.g. https://www.googleapis.com/compute/alpha/projects/PROJECT/ regions/REGION/subnetworks/SUBNETWORK

setSubnetwork

void setSubnetwork(java.lang.String value)

getWorkerMachineType

java.lang.String getWorkerMachineType()

Machine type to create Dataflow worker VMs as.

See GCE machine types for a list of valid options.

If unset, the Dataflow service will choose a reasonable default.

setWorkerMachineType

void setWorkerMachineType(java.lang.String value)

getFilesToStage

java.util.List<java.lang.String> getFilesToStage()

List of local files to make available to workers.

Files are placed on the worker's classpath.

The default value is the list of jars from the main program's classpath.

setFilesToStage

void setFilesToStage(java.util.List<java.lang.String> value)

getWorkerDiskType

java.lang.String getWorkerDiskType()

Specifies what type of persistent disk is used. The value is a full disk type resource, e.g., compute.googleapis.com/projects//zones//diskTypes/pd-ssd. For more information, see the API reference documentation for DiskTypes.

setWorkerDiskType

void setWorkerDiskType(java.lang.String value)

getUsePublicIps

@Experimental
@Nullable java.lang.Boolean getUsePublicIps()

Specifies whether worker pools should be started with public IP addresses.

WARNING: This feature is experimental. You must be allowlisted to use it.

setUsePublicIps

void setUsePublicIps(@Nullable java.lang.Boolean value)

getMinCpuPlatform

@Nullable java.lang.String getMinCpuPlatform()

Specifies a Minimum CPU platform for VM instances.

More details see Specifying Pipeline Execution Parameters.

setMinCpuPlatform

void setMinCpuPlatform(java.lang.String minCpuPlatform)