Package org.apache.beam.runners.spark

Interface SparkCommonPipelineOptions

All Superinterfaces:

ApplicationNameOptions, FileStagingOptions, HasDisplayData, PipelineOptions, StreamingOptions

All Known Subinterfaces:

SparkContextOptions, SparkPipelineOptions, SparkPortableStreamingPipelineOptions, SparkStructuredStreamingPipelineOptions, TestSparkPipelineOptions

public interface SparkCommonPipelineOptions
extends PipelineOptions, StreamingOptions, ApplicationNameOptions, FileStagingOptions

Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, and other user-related knobs.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	SparkCommonPipelineOptions.StorageLevelFactory	Returns Spark's default storage level for the Dataset or RDD API based on the respective runner.
static class	SparkCommonPipelineOptions.TmpCheckpointDirFactory	Returns the default checkpoint directory of /tmp/${job.name}.

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DEFAULT_MASTER_URL

Method Summary

Modifier and Type	Method	Description
String	getCheckpointDir()
Boolean	getEnableSparkMetricSinks()
Boolean	getPreferGroupByKeyToHandleHugeValues()
String	getSparkMaster()
String	getStorageLevel()
static void	prepareFilesToStage(SparkCommonPipelineOptions options)	Classpath contains non jar files (eg.
void	setCheckpointDir(String checkpointDir)
void	setEnableSparkMetricSinks(Boolean enableSparkMetricSinks)
void	setPreferGroupByKeyToHandleHugeValues(Boolean preferGroupByKeyToHandleHugeValues)
void	setSparkMaster(String master)
void	setStorageLevel(String storageLevel)

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.FileStagingOptions

getFilesToStage, setFilesToStage

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getGbek, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setGbek, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

getUpdateCompatibilityVersion, isStreaming, setStreaming, setUpdateCompatibilityVersion

Field Details

DEFAULT_MASTER_URL

static final String DEFAULT_MASTER_URL

See Also:

• Constant Field Values

Method Details

getSparkMaster

@String("local[4]")
String getSparkMaster()

setSparkMaster

void setSparkMaster(String master)

getCheckpointDir

@InstanceFactory(TmpCheckpointDirFactory.class)
String getCheckpointDir()

setCheckpointDir

void setCheckpointDir(String checkpointDir)

getStorageLevel

@InstanceFactory(StorageLevelFactory.class)
String getStorageLevel()

setStorageLevel

void setStorageLevel(String storageLevel)

getEnableSparkMetricSinks

@Boolean(true)
Boolean getEnableSparkMetricSinks()

setEnableSparkMetricSinks

void setEnableSparkMetricSinks(Boolean enableSparkMetricSinks)

getPreferGroupByKeyToHandleHugeValues

@Boolean(false)
Boolean getPreferGroupByKeyToHandleHugeValues()

setPreferGroupByKeyToHandleHugeValues

void setPreferGroupByKeyToHandleHugeValues(Boolean preferGroupByKeyToHandleHugeValues)

prepareFilesToStage

@Internal
static void prepareFilesToStage(SparkCommonPipelineOptions options)

Classpath contains non jar files (eg. directories with .class files or empty directories) will cause exception in running log. Though the SparkContext can handle this when running in local master, it's better not to include non-jars files in classpath.