org.apache.beam.runners.spark

Interface SparkCommonPipelineOptions

All Superinterfaces:

ApplicationNameOptions, HasDisplayData, PipelineOptions, StreamingOptions

All Known Subinterfaces:

SparkContextOptions, SparkPipelineOptions, SparkStructuredStreamingPipelineOptions, TestSparkPipelineOptions

public interface SparkCommonPipelineOptions
extends PipelineOptions, StreamingOptions, ApplicationNameOptions

Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, and other user-related knobs.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	SparkCommonPipelineOptions.TmpCheckpointDirFactory Returns the default checkpoint directory of /tmp/${job.name}.

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_MASTER_URL

Method Summary

Modifier and Type	Method and Description
java.lang.String	getCheckpointDir()
java.lang.Boolean	getEnableSparkMetricSinks()
java.util.List<java.lang.String>	getFilesToStage() List of local files to make available to workers.
java.lang.String	getSparkMaster()
void	setCheckpointDir(java.lang.String checkpointDir)
void	setEnableSparkMetricSinks(java.lang.Boolean enableSparkMetricSinks)
void	setFilesToStage(java.util.List<java.lang.String> value)
void	setSparkMaster(java.lang.String master)

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

isStreaming, setStreaming

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Field Detail

DEFAULT_MASTER_URL

static final java.lang.String DEFAULT_MASTER_URL

See Also:

Constant Field Values

Method Detail

getSparkMaster

@Default.String(value="local[4]")
java.lang.String getSparkMaster()

setSparkMaster

void setSparkMaster(java.lang.String master)

getCheckpointDir

@Default.InstanceFactory(value=SparkCommonPipelineOptions.TmpCheckpointDirFactory.class)
java.lang.String getCheckpointDir()

setCheckpointDir

void setCheckpointDir(java.lang.String checkpointDir)

getFilesToStage

java.util.List<java.lang.String> getFilesToStage()

List of local files to make available to workers.

Jars are placed on the worker's classpath.

The default value is the list of jars from the main program's classpath.

setFilesToStage

void setFilesToStage(java.util.List<java.lang.String> value)

getEnableSparkMetricSinks

@Default.Boolean(value=true)
java.lang.Boolean getEnableSparkMetricSinks()

setEnableSparkMetricSinks

void setEnableSparkMetricSinks(java.lang.Boolean enableSparkMetricSinks)