Package org.apache.beam.runners.spark

Interface SparkContextOptions

All Superinterfaces:
ApplicationNameOptions, FileStagingOptions, HasDisplayData, PipelineOptions, SparkCommonPipelineOptions, SparkPipelineOptions, StreamingOptions
public interface SparkContextOptions extends SparkPipelineOptions
A custom PipelineOptions to work with properties related to JavaSparkContext.

This can only be used programmatically (as opposed to passing command line arguments), since the properties here are context-aware and should not be propagated to workers.

Separating this from SparkPipelineOptions is needed so the context-aware properties, which link to Spark dependencies, won't be scanned by PipelineOptions reflective instantiation. Note that SparkContextOptions is not registered with SparkRunnerRegistrar.

Note: It's recommended to use SparkContextFactory.setProvidedSparkContext(JavaSparkContext) instead of setProvidedSparkContext(JavaSparkContext) for testing. When using @TestPipeline any provided JavaSparkContext via SparkContextOptions is dropped.

  • Method Details

    • getProvidedSparkContext

      org.apache.spark.api.java.JavaSparkContext getProvidedSparkContext()

    • setProvidedSparkContext

      void setProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext jsc)

    • getListeners

      @InstanceFactory(EmptyListenersList.class) List<org.apache.spark.streaming.api.java.JavaStreamingListener> getListeners()

    • setListeners

      void setListeners(List<org.apache.spark.streaming.api.java.JavaStreamingListener> listeners)