org.apache.beam.runners.spark

Interface SparkPipelineOptions

All Superinterfaces:

ApplicationNameOptions, HasDisplayData, PipelineOptions, SparkCommonPipelineOptions, StreamingOptions

All Known Subinterfaces:

SparkContextOptions, SparkPortableStreamingPipelineOptions, TestSparkPipelineOptions

public interface SparkPipelineOptions
extends SparkCommonPipelineOptions

Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, batch-interval, and other user-related knobs.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

SparkCommonPipelineOptions.TmpCheckpointDirFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

DEFAULT_MASTER_URL

Method Summary

Modifier and Type	Method and Description
java.lang.Long	getBatchIntervalMillis()
java.lang.Long	getBundleSize()
java.lang.Long	getCheckpointDurationMillis()
java.lang.Long	getMaxRecordsPerBatch()
java.lang.Long	getMinReadTimeMillis()
java.lang.Double	getReadTimePercentage()
java.lang.String	getStorageLevel()
boolean	getUsesProvidedSparkContext()
boolean	isCacheDisabled()
static boolean	isLocalSparkMaster(SparkPipelineOptions options) Detects if the pipeline is run in spark local mode.
static void	prepareFilesToStage(SparkPipelineOptions options) Classpath contains non jar files (eg.
void	setBatchIntervalMillis(java.lang.Long batchInterval)
void	setBundleSize(java.lang.Long value)
void	setCacheDisabled(boolean value)
void	setCheckpointDurationMillis(java.lang.Long durationMillis)
void	setMaxRecordsPerBatch(java.lang.Long maxRecordsPerBatch)
void	setMinReadTimeMillis(java.lang.Long minReadTimeMillis)
void	setReadTimePercentage(java.lang.Double readTimePercentage)
void	setStorageLevel(java.lang.String storageLevel)
void	setUsesProvidedSparkContext(boolean value)

Methods inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

getCheckpointDir, getEnableSparkMetricSinks, getFilesToStage, getSparkMaster, setCheckpointDir, setEnableSparkMetricSinks, setFilesToStage, setSparkMaster

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

isStreaming, setStreaming

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Method Detail

getBatchIntervalMillis

@Default.Long(value=500L)
java.lang.Long getBatchIntervalMillis()

setBatchIntervalMillis

void setBatchIntervalMillis(java.lang.Long batchInterval)

getStorageLevel

@Default.String(value="MEMORY_ONLY")
java.lang.String getStorageLevel()

setStorageLevel

void setStorageLevel(java.lang.String storageLevel)

getMinReadTimeMillis

@Default.Long(value=200L)
java.lang.Long getMinReadTimeMillis()

setMinReadTimeMillis

void setMinReadTimeMillis(java.lang.Long minReadTimeMillis)

getMaxRecordsPerBatch

@Default.Long(value=-1L)
java.lang.Long getMaxRecordsPerBatch()

setMaxRecordsPerBatch

void setMaxRecordsPerBatch(java.lang.Long maxRecordsPerBatch)

getReadTimePercentage

@Default.Double(value=0.1)
java.lang.Double getReadTimePercentage()

setReadTimePercentage

void setReadTimePercentage(java.lang.Double readTimePercentage)

getCheckpointDurationMillis

@Default.Long(value=-1L)
java.lang.Long getCheckpointDurationMillis()

setCheckpointDurationMillis

void setCheckpointDurationMillis(java.lang.Long durationMillis)

getBundleSize

@Default.Long(value=0L)
java.lang.Long getBundleSize()

setBundleSize

@Experimental
void setBundleSize(java.lang.Long value)

getUsesProvidedSparkContext

@Default.Boolean(value=false)
boolean getUsesProvidedSparkContext()

setUsesProvidedSparkContext

void setUsesProvidedSparkContext(boolean value)

isCacheDisabled

@Default.Boolean(value=false)
boolean isCacheDisabled()

setCacheDisabled

void setCacheDisabled(boolean value)

isLocalSparkMaster

@Internal
static boolean isLocalSparkMaster(SparkPipelineOptions options)

Detects if the pipeline is run in spark local mode.

prepareFilesToStage

@Internal
static void prepareFilesToStage(SparkPipelineOptions options)

Classpath contains non jar files (eg. directories with .class files or empty directories) will cause exception in running log. Though the SparkContext can handle this when running in local master, it's better not to include non-jars files in classpath.