org.apache.beam.runners.spark

Interface SparkCommonPipelineOptions

All Superinterfaces:

ApplicationNameOptions, FileStagingOptions, HasDisplayData, PipelineOptions, StreamingOptions

All Known Subinterfaces:

SparkContextOptions, SparkPipelineOptions, SparkPortableStreamingPipelineOptions, SparkStructuredStreamingPipelineOptions, TestSparkPipelineOptions

public interface SparkCommonPipelineOptions
extends PipelineOptions, StreamingOptions, ApplicationNameOptions, FileStagingOptions

Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, and other user-related knobs.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	SparkCommonPipelineOptions.StorageLevelFactory Returns Spark's default storage level for the Dataset or RDD API based on the respective runner.
static class	SparkCommonPipelineOptions.TmpCheckpointDirFactory Returns the default checkpoint directory of /tmp/${job.name}.

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_MASTER_URL

Method Summary

Modifier and Type	Method and Description
java.lang.String	getCheckpointDir()
java.lang.Boolean	getEnableSparkMetricSinks()
java.lang.String	getSparkMaster()
java.lang.String	getStorageLevel()
static void	prepareFilesToStage(SparkCommonPipelineOptions options) Classpath contains non jar files (eg.
void	setCheckpointDir(java.lang.String checkpointDir)
void	setEnableSparkMetricSinks(java.lang.Boolean enableSparkMetricSinks)
void	setSparkMaster(java.lang.String master)
void	setStorageLevel(java.lang.String storageLevel)

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

getUpdateCompatibilityVersion, isStreaming, setStreaming, setUpdateCompatibilityVersion, updateCompatibilityVersionLessThan

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface org.apache.beam.sdk.options.FileStagingOptions

getFilesToStage, setFilesToStage

Field Detail

DEFAULT_MASTER_URL

static final java.lang.String DEFAULT_MASTER_URL

See Also:

Constant Field Values

Method Detail

getSparkMaster

@Default.String(value="local[4]")
java.lang.String getSparkMaster()

setSparkMaster

void setSparkMaster(java.lang.String master)

getCheckpointDir

@Default.InstanceFactory(value=SparkCommonPipelineOptions.TmpCheckpointDirFactory.class)
java.lang.String getCheckpointDir()

setCheckpointDir

void setCheckpointDir(java.lang.String checkpointDir)

getStorageLevel

@Default.InstanceFactory(value=SparkCommonPipelineOptions.StorageLevelFactory.class)
java.lang.String getStorageLevel()

setStorageLevel

void setStorageLevel(java.lang.String storageLevel)

getEnableSparkMetricSinks

@Default.Boolean(value=true)
java.lang.Boolean getEnableSparkMetricSinks()

setEnableSparkMetricSinks

void setEnableSparkMetricSinks(java.lang.Boolean enableSparkMetricSinks)

prepareFilesToStage

@Internal
static void prepareFilesToStage(SparkCommonPipelineOptions options)

Classpath contains non jar files (eg. directories with .class files or empty directories) will cause exception in running log. Though the SparkContext can handle this when running in local master, it's better not to include non-jars files in classpath.