org.apache.beam.runners.spark.structuredstreaming

Interface SparkStructuredStreamingPipelineOptions

All Superinterfaces:

ApplicationNameOptions, FileStagingOptions, HasDisplayData, PipelineOptions, SparkCommonPipelineOptions, StreamingOptions

public interface SparkStructuredStreamingPipelineOptions
extends SparkCommonPipelineOptions

Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, and other user-related knobs.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

SparkCommonPipelineOptions.StorageLevelFactory, SparkCommonPipelineOptions.TmpCheckpointDirFactory

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled, PipelineOptions.DirectRunner, PipelineOptions.JobNameFactory, PipelineOptions.UserAgentFactory

Field Summary

Fields inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

DEFAULT_MASTER_URL

Method Summary

Modifier and Type	Method and Description
boolean	getTestMode() Set to true to run the job in test mode.
boolean	getUseActiveSparkSession()
void	setTestMode(boolean testMode)
void	setUseActiveSparkSession(boolean value)

Methods inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

getCheckpointDir, getEnableSparkMetricSinks, getSparkMaster, getStorageLevel, prepareFilesToStage, setCheckpointDir, setEnableSparkMetricSinks, setSparkMaster, setStorageLevel

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

getUpdateCompatibilityVersion, isStreaming, setStreaming, setUpdateCompatibilityVersion, updateCompatibilityVersionLessThan

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface org.apache.beam.sdk.options.FileStagingOptions

getFilesToStage, setFilesToStage

Method Detail

getTestMode

@Default.Boolean(value=false)
boolean getTestMode()

Set to true to run the job in test mode.

setTestMode

void setTestMode(boolean testMode)

getUseActiveSparkSession

@Default.Boolean(value=false)
boolean getUseActiveSparkSession()

setUseActiveSparkSession

void setUseActiveSparkSession(boolean value)