public interface SparkCommonPipelineOptions extends PipelineOptions, StreamingOptions, ApplicationNameOptions
PipelineOptions
handles Spark execution-related configurations, such as the
master address, and other user-related knobs.Modifier and Type | Interface and Description |
---|---|
static class |
SparkCommonPipelineOptions.TmpCheckpointDirFactory
Returns the default checkpoint directory of /tmp/${job.name}.
|
PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
DEFAULT_MASTER_URL |
Modifier and Type | Method and Description |
---|---|
java.lang.String |
getCheckpointDir() |
java.lang.Boolean |
getEnableSparkMetricSinks() |
java.util.List<java.lang.String> |
getFilesToStage()
List of local files to make available to workers.
|
java.lang.String |
getSparkMaster() |
void |
setCheckpointDir(java.lang.String checkpointDir) |
void |
setEnableSparkMetricSinks(java.lang.Boolean enableSparkMetricSinks) |
void |
setFilesToStage(java.util.List<java.lang.String> value) |
void |
setSparkMaster(java.lang.String master) |
isStreaming, setStreaming
getAppName, setAppName
as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent
populateDisplayData
static final java.lang.String DEFAULT_MASTER_URL
@Default.String(value="local[4]") java.lang.String getSparkMaster()
void setSparkMaster(java.lang.String master)
@Default.InstanceFactory(value=SparkCommonPipelineOptions.TmpCheckpointDirFactory.class) java.lang.String getCheckpointDir()
void setCheckpointDir(java.lang.String checkpointDir)
java.util.List<java.lang.String> getFilesToStage()
Jars are placed on the worker's classpath.
The default value is the list of jars from the main program's classpath.
void setFilesToStage(java.util.List<java.lang.String> value)
@Default.Boolean(value=true) java.lang.Boolean getEnableSparkMetricSinks()
void setEnableSparkMetricSinks(java.lang.Boolean enableSparkMetricSinks)