org.apache.beam.runners.dataflow.options

Interface DataflowPipelineDebugOptions

All Superinterfaces:

ExperimentalOptions, HasDisplayData, PipelineOptions

All Known Subinterfaces:

DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

@Hidden
public interface DataflowPipelineDebugOptions
extends ExperimentalOptions, PipelineOptions

Internal. Options used to control execution of the Dataflow SDK for debugging and testing purposes.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataflowPipelineDebugOptions.DataflowClientFactory Returns the default Dataflow client built from the passed in PipelineOptions.
static class	DataflowPipelineDebugOptions.StagerFactory Creates a Stager object using the class specified in getStagerClass().

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

STATE_CACHE_SIZE, STATE_SAMPLING_PERIOD_MILLIS

Method Summary

Modifier and Type	Method and Description
java.lang.String	getApiRootUrl() The root URL for the Dataflow API.
Dataflow	getDataflowClient() An instance of the Dataflow client.
java.lang.String	getDataflowEndpoint() Dataflow endpoint to use.
java.lang.String	getDataflowJobFile() The path to write the translated Dataflow job specification out to at job submission time.
boolean	getDumpHeapOnOOM() If true, save a heap dump before killing a thread or process which is GC thrashing or out of memory.
java.lang.Double	getGCThrashingPercentagePerPeriod() The GC thrashing threshold percentage.
int	getJfrRecordingDurationSec()
int	getMaxBundlesFromWindmillOutstanding() Maximum number of bundles outstanding from windmill before the worker stops requesting.
long	getMaxBytesFromWindmillOutstanding() Maximum number of bytes outstanding from windmill before the worker stops requesting.
int	getNumberOfWorkerHarnessThreads() Number of threads to use on the Dataflow worker harness.
java.lang.String	getOverrideWindmillBinary() Custom windmill_main binary to use with the streaming runner.
java.lang.Integer	getReaderCacheTimeoutSec() The amount of time before UnboundedReaders are considered idle and closed during streaming execution.
boolean	getRecordJfrOnGcThrashing() If true, save a JFR profile when GC thrashing is first detected.
java.lang.String	getSaveHeapDumpsToGcsPath() CAUTION: This option implies dumpHeapOnOOM, and has similar caveats.
java.lang.String	getSdkHarnessContainerImageOverrides() Overrides for SDK harness container images.
Stager	getStager() The resource stager instance that should be used to stage resources.
java.lang.Class<? extends Stager>	getStagerClass() The class responsible for staging resources to be accessible by workers during job execution.
java.util.Map<java.lang.String,java.lang.String>	getTransformNameMapping() Mapping of old PTransform names to new ones, specified as JSON {"oldName":"newName",...} .
java.lang.Integer	getUnboundedReaderMaxElements() The max elements read from an UnboundedReader before checkpointing.
java.lang.Integer	getUnboundedReaderMaxReadTimeSec() The max amount of time an UnboundedReader is consumed before checkpointing.
java.lang.Integer	getUnboundedReaderMaxWaitForElementsMs() The max amount of time waiting for elements when reading from UnboundedReader.
java.lang.String	getWindmillServiceEndpoint() Custom windmill service endpoint.
int	getWindmillServicePort()
java.lang.Integer	getWorkerCacheMb() The size of the worker's in-memory cache, in megabytes.
void	setApiRootUrl(java.lang.String value)
void	setDataflowClient(Dataflow value)
void	setDataflowEndpoint(java.lang.String value)
void	setDataflowJobFile(java.lang.String value)
void	setDumpHeapOnOOM(boolean dumpHeapBeforeExit)
void	setGCThrashingPercentagePerPeriod(java.lang.Double value)
void	setJfrRecordingDurationSec(int value)
void	setMaxBundlesFromWindmillOutstanding(int value)
void	setMaxBytesFromWindmillOutstanding(long value)
void	setNumberOfWorkerHarnessThreads(int value)
void	setOverrideWindmillBinary(java.lang.String value)
void	setReaderCacheTimeoutSec(java.lang.Integer value)
void	setRecordJfrOnGcThrashing(boolean value)
void	setSaveHeapDumpsToGcsPath(java.lang.String gcsPath)
void	setSdkHarnessContainerImageOverrides(java.lang.String value)
void	setStager(Stager stager)
void	setStagerClass(java.lang.Class<? extends Stager> stagerClass)
void	setTransformNameMapping(java.util.Map<java.lang.String,java.lang.String> value)
void	setUnboundedReaderMaxElements(java.lang.Integer value)
void	setUnboundedReaderMaxReadTimeSec(java.lang.Integer value)
void	setUnboundedReaderMaxWaitForElementsMs(java.lang.Integer value)
void	setWindmillServiceEndpoint(java.lang.String value)
void	setWindmillServicePort(int value)
void	setWorkerCacheMb(java.lang.Integer value)

Methods inherited from interface org.apache.beam.sdk.options.ExperimentalOptions

addExperiment, getExperiments, getExperimentValue, hasExperiment, setExperiments

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Method Detail

getApiRootUrl

@Default.String(value="https://dataflow.googleapis.com/")
java.lang.String getApiRootUrl()

The root URL for the Dataflow API. dataflowEndpoint can override this value if it contains an absolute URL, otherwise apiRootUrl will be combined with dataflowEndpoint to generate the full URL to communicate with the Dataflow API.

setApiRootUrl

void setApiRootUrl(java.lang.String value)

getDataflowEndpoint

@Default.String(value="")
java.lang.String getDataflowEndpoint()

Dataflow endpoint to use.

Defaults to the current version of the Google Cloud Dataflow API, at the time the current SDK version was released.

If the string contains "://", then this is treated as a URL, otherwise getApiRootUrl() is used as the root URL.

setDataflowEndpoint

void setDataflowEndpoint(java.lang.String value)

getDataflowJobFile

java.lang.String getDataflowJobFile()

The path to write the translated Dataflow job specification out to at job submission time. The Dataflow job specification will be represented in JSON format.

setDataflowJobFile

void setDataflowJobFile(java.lang.String value)

getStagerClass

@Default.Class(value=GcsStager.class)
java.lang.Class<? extends Stager> getStagerClass()

The class responsible for staging resources to be accessible by workers during job execution. If stager has not been set explicitly, an instance of this class will be created and used as the resource stager.

setStagerClass

void setStagerClass(java.lang.Class<? extends Stager> stagerClass)

getStager

@Default.InstanceFactory(value=DataflowPipelineDebugOptions.StagerFactory.class)
Stager getStager()

The resource stager instance that should be used to stage resources. If no stager has been set explicitly, the default is to use the instance factory that constructs a resource stager based upon the currently set stagerClass.

setStager

void setStager(Stager stager)

getDataflowClient

@Default.InstanceFactory(value=DataflowPipelineDebugOptions.DataflowClientFactory.class)
Dataflow getDataflowClient()

An instance of the Dataflow client. Defaults to creating a Dataflow client using the current set of options.

setDataflowClient

void setDataflowClient(Dataflow value)

getTransformNameMapping

java.util.Map<java.lang.String,java.lang.String> getTransformNameMapping()

Mapping of old PTransform names to new ones, specified as JSON {"oldName":"newName",...} . To mark a transform as deleted, make newName the empty string.

setTransformNameMapping

void setTransformNameMapping(java.util.Map<java.lang.String,java.lang.String> value)

getOverrideWindmillBinary

java.lang.String getOverrideWindmillBinary()

Custom windmill_main binary to use with the streaming runner.

setOverrideWindmillBinary

void setOverrideWindmillBinary(java.lang.String value)

getWindmillServiceEndpoint

java.lang.String getWindmillServiceEndpoint()

Custom windmill service endpoint.

setWindmillServiceEndpoint

void setWindmillServiceEndpoint(java.lang.String value)

getWindmillServicePort

@Default.Integer(value=443)
int getWindmillServicePort()

setWindmillServicePort

void setWindmillServicePort(int value)

getNumberOfWorkerHarnessThreads

int getNumberOfWorkerHarnessThreads()

Number of threads to use on the Dataflow worker harness. If left unspecified, the Dataflow service will compute an appropriate number of threads to use.

setNumberOfWorkerHarnessThreads

void setNumberOfWorkerHarnessThreads(int value)

getMaxBundlesFromWindmillOutstanding

@Default.Integer(value=0)
int getMaxBundlesFromWindmillOutstanding()

Maximum number of bundles outstanding from windmill before the worker stops requesting.

If <= 0, use the default value of 100 + getNumberOfWorkerHarnessThreads()

setMaxBundlesFromWindmillOutstanding

void setMaxBundlesFromWindmillOutstanding(int value)

getMaxBytesFromWindmillOutstanding

@Default.Long(value=0L)
long getMaxBytesFromWindmillOutstanding()

Maximum number of bytes outstanding from windmill before the worker stops requesting.

If <= 0, use the default value of 50% of jvm memory.

setMaxBytesFromWindmillOutstanding

void setMaxBytesFromWindmillOutstanding(long value)

getDumpHeapOnOOM

boolean getDumpHeapOnOOM()

If true, save a heap dump before killing a thread or process which is GC thrashing or out of memory. The location of the heap file will either be echoed back to the user, or the user will be given the opportunity to download the heap file.

CAUTION: Heap dumps can of comparable size to the default boot disk. Consider increasing the boot disk size before setting this flag to true.

setDumpHeapOnOOM

void setDumpHeapOnOOM(boolean dumpHeapBeforeExit)

getRecordJfrOnGcThrashing

boolean getRecordJfrOnGcThrashing()

If true, save a JFR profile when GC thrashing is first detected. The profile will run for the amount of time set by --jfrRecordingDurationSec, or 60 seconds by default.

Note, JFR profiles are only supported on java 9 and up.

setRecordJfrOnGcThrashing

void setRecordJfrOnGcThrashing(boolean value)

getJfrRecordingDurationSec

@Default.Integer(value=60)
int getJfrRecordingDurationSec()

setJfrRecordingDurationSec

void setJfrRecordingDurationSec(int value)

getGCThrashingPercentagePerPeriod

@Default.Double(value=50.0)
java.lang.Double getGCThrashingPercentagePerPeriod()

The GC thrashing threshold percentage. A given period of time is considered "thrashing" if this percentage of CPU time is spent in garbage collection. Dataflow will force fail tasks after sustained periods of thrashing.

If 100 is given as the value, MemoryMonitor will be disabled.

setGCThrashingPercentagePerPeriod

void setGCThrashingPercentagePerPeriod(java.lang.Double value)

getWorkerCacheMb

@Default.Integer(value=100)
java.lang.Integer getWorkerCacheMb()

The size of the worker's in-memory cache, in megabytes.

Currently, this cache is used for storing read values of side inputs. as well as the state for streaming jobs.

setWorkerCacheMb

void setWorkerCacheMb(java.lang.Integer value)

getReaderCacheTimeoutSec

@Default.Integer(value=60)
java.lang.Integer getReaderCacheTimeoutSec()

The amount of time before UnboundedReaders are considered idle and closed during streaming execution.

setReaderCacheTimeoutSec

void setReaderCacheTimeoutSec(java.lang.Integer value)

getUnboundedReaderMaxReadTimeSec

@Default.Integer(value=10)
java.lang.Integer getUnboundedReaderMaxReadTimeSec()

The max amount of time an UnboundedReader is consumed before checkpointing.

setUnboundedReaderMaxReadTimeSec

void setUnboundedReaderMaxReadTimeSec(java.lang.Integer value)

getUnboundedReaderMaxElements

@Default.Integer(value=10000)
java.lang.Integer getUnboundedReaderMaxElements()

The max elements read from an UnboundedReader before checkpointing.

setUnboundedReaderMaxElements

void setUnboundedReaderMaxElements(java.lang.Integer value)

getUnboundedReaderMaxWaitForElementsMs

@Default.Integer(value=1000)
java.lang.Integer getUnboundedReaderMaxWaitForElementsMs()

The max amount of time waiting for elements when reading from UnboundedReader.

setUnboundedReaderMaxWaitForElementsMs

void setUnboundedReaderMaxWaitForElementsMs(java.lang.Integer value)

getSaveHeapDumpsToGcsPath

java.lang.String getSaveHeapDumpsToGcsPath()

CAUTION: This option implies dumpHeapOnOOM, and has similar caveats. Specifically, heap dumps can of comparable size to the default boot disk. Consider increasing the boot disk size before setting this flag to true.

setSaveHeapDumpsToGcsPath

void setSaveHeapDumpsToGcsPath(java.lang.String gcsPath)

getSdkHarnessContainerImageOverrides

java.lang.String getSdkHarnessContainerImageOverrides()

Overrides for SDK harness container images.

setSdkHarnessContainerImageOverrides

void setSdkHarnessContainerImageOverrides(java.lang.String value)