org.apache.beam.runners.spark

Interface SparkContextOptions

All Superinterfaces:

ApplicationNameOptions, FileStagingOptions, HasDisplayData, PipelineOptions, SparkCommonPipelineOptions, SparkPipelineOptions, StreamingOptions

public interface SparkContextOptions
extends SparkPipelineOptions

A custom PipelineOptions to work with properties related to JavaSparkContext.

This can only be used programmatically (as opposed to passing command line arguments), since the properties here are context-aware and should not be propagated to workers.

Separating this from SparkPipelineOptions is needed so the context-aware properties, which link to Spark dependencies, won't be scanned by PipelineOptions reflective instantiation. Note that SparkContextOptions is not registered with SparkRunnerRegistrar.

Note: It's recommended to use org.apache.beam.runners.spark.translation.SparkContextFactory#setProvidedSparkContext(JavaSparkContext) instead of setProvidedSparkContext(JavaSparkContext) for testing. When using @TestPipeline any provided JavaSparkContext via SparkContextOptions is dropped.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	SparkContextOptions.EmptyListenersList Returns an empty list, to avoid handling null.

Nested classes/interfaces inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

SparkCommonPipelineOptions.StorageLevelFactory, SparkCommonPipelineOptions.TmpCheckpointDirFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

DEFAULT_MASTER_URL

Method Summary

Modifier and Type	Method and Description
java.util.List<org.apache.spark.streaming.api.java.JavaStreamingListener>	getListeners()
org.apache.spark.api.java.JavaSparkContext	getProvidedSparkContext()
void	setListeners(java.util.List<org.apache.spark.streaming.api.java.JavaStreamingListener> listeners)
void	setProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext jsc)

Methods inherited from interface org.apache.beam.runners.spark.SparkPipelineOptions

getBatchIntervalMillis, getBundleSize, getCheckpointDurationMillis, getMaxRecordsPerBatch, getMinReadTimeMillis, getReadTimePercentage, getUsesProvidedSparkContext, isCacheDisabled, setBatchIntervalMillis, setBundleSize, setCacheDisabled, setCheckpointDurationMillis, setMaxRecordsPerBatch, setMinReadTimeMillis, setReadTimePercentage, setUsesProvidedSparkContext

Methods inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

getCheckpointDir, getEnableSparkMetricSinks, getSparkMaster, getStorageLevel, prepareFilesToStage, setCheckpointDir, setEnableSparkMetricSinks, setSparkMaster, setStorageLevel

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

getUpdateCompatibilityVersion, isStreaming, setStreaming, setUpdateCompatibilityVersion, updateCompatibilityVersionLessThan

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface org.apache.beam.sdk.options.FileStagingOptions

getFilesToStage, setFilesToStage

Method Detail

getProvidedSparkContext

org.apache.spark.api.java.JavaSparkContext getProvidedSparkContext()

setProvidedSparkContext

void setProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext jsc)

getListeners

@Default.InstanceFactory(value=SparkContextOptions.EmptyListenersList.class)
java.util.List<org.apache.spark.streaming.api.java.JavaStreamingListener> getListeners()

setListeners

void setListeners(java.util.List<org.apache.spark.streaming.api.java.JavaStreamingListener> listeners)