org.apache.beam.runners.spark.structuredstreaming

Class SparkStructuredStreamingRunner

java.lang.Object

org.apache.beam.sdk.PipelineRunner<SparkStructuredStreamingPipelineResult>

org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner

public final class SparkStructuredStreamingRunner
extends PipelineRunner<SparkStructuredStreamingPipelineResult>

SparkStructuredStreamingRunner is based on spark structured streaming framework and is no more based on RDD/DStream API. See https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html It is still experimental, its coverage of the Beam model is partial. The SparkStructuredStreamingRunner translate operations defined on a pipeline to a representation executable by Spark, and then submitting the job to Spark to be executed. If we wanted to run a Beam pipeline with the default options of a single threaded spark instance in local mode, we would do the following:

Pipeline p = [logic for pipeline creation] SparkStructuredStreamingPipelineResult result = (SparkStructuredStreamingPipelineResult) p.run();

To create a pipeline runner to run against a different spark cluster, with a custom master url we would do the following:

Pipeline p = [logic for pipeline creation] SparkStructuredStreamingPipelineOptions options = SparkPipelineOptionsFactory.create(); options.setSparkMaster("spark://host:port"); SparkStructuredStreamingPipelineResult result = (SparkStructuredStreamingPipelineResult) p.run();

Method Summary

Modifier and Type	Method and Description
static SparkStructuredStreamingRunner	create() Creates and returns a new SparkStructuredStreamingRunner with default options.
static SparkStructuredStreamingRunner	create(SparkStructuredStreamingPipelineOptions options) Creates and returns a new SparkStructuredStreamingRunner with specified options.
static SparkStructuredStreamingRunner	fromOptions(PipelineOptions options) Creates and returns a new SparkStructuredStreamingRunner with specified options.
static void	initAccumulators(SparkStructuredStreamingPipelineOptions opts, org.apache.spark.api.java.JavaSparkContext jsc) Init Metrics/Aggregators accumulators.
SparkStructuredStreamingPipelineResult	run(Pipeline pipeline) Processes the given Pipeline, potentially asynchronously, returning a runner-specific type of result.

Methods inherited from class org.apache.beam.sdk.PipelineRunner

run, run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static SparkStructuredStreamingRunner create()

Creates and returns a new SparkStructuredStreamingRunner with default options. In particular, against a spark instance running in local mode.

Returns:

A pipeline runner with default options.

create

public static SparkStructuredStreamingRunner create(SparkStructuredStreamingPipelineOptions options)

Creates and returns a new SparkStructuredStreamingRunner with specified options.

Parameters:

options - The SparkStructuredStreamingPipelineOptions to use when executing the job.

Returns:

A pipeline runner that will execute with specified options.

fromOptions

public static SparkStructuredStreamingRunner fromOptions(PipelineOptions options)

Creates and returns a new SparkStructuredStreamingRunner with specified options.

Parameters:

options - The PipelineOptions to use when executing the job.

Returns:

A pipeline runner that will execute with specified options.

run

public SparkStructuredStreamingPipelineResult run(Pipeline pipeline)

Description copied from class: PipelineRunner

Processes the given Pipeline, potentially asynchronously, returning a runner-specific type of result.

Specified by:

run in class PipelineRunner<SparkStructuredStreamingPipelineResult>

initAccumulators

public static void initAccumulators(SparkStructuredStreamingPipelineOptions opts,
                                    org.apache.spark.api.java.JavaSparkContext jsc)

Init Metrics/Aggregators accumulators. This method is idempotent.