Package org.apache.beam.runners.spark.translation

Class SparkStreamingPortablePipelineTranslator

java.lang.Object

org.apache.beam.runners.spark.translation.SparkStreamingPortablePipelineTranslator

All Implemented Interfaces:

SparkPortablePipelineTranslator<SparkStreamingTranslationContext>

public class SparkStreamingPortablePipelineTranslator
extends Object
implements SparkPortablePipelineTranslator<SparkStreamingTranslationContext>

Translates an unbounded portable pipeline into a Spark job.

Constructor Summary

Constructors

Constructor	Description
SparkStreamingPortablePipelineTranslator()

Method Summary

Modifier and Type	Method	Description
SparkStreamingTranslationContext	createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, JobInfo jobInfo)
Set<String>	knownUrns()
void	translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline, SparkStreamingTranslationContext context)	Translates pipeline from Beam into the Spark context.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SparkStreamingPortablePipelineTranslator

public SparkStreamingPortablePipelineTranslator()

Method Details

knownUrns

public Set<String> knownUrns()

Specified by:

knownUrns in interface SparkPortablePipelineTranslator<SparkStreamingTranslationContext>

translate

public void translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline,
 SparkStreamingTranslationContext context)

Translates pipeline from Beam into the Spark context.

Specified by:

translate in interface SparkPortablePipelineTranslator<SparkStreamingTranslationContext>

createTranslationContext

public SparkStreamingTranslationContext createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc,
 SparkPipelineOptions options,
 JobInfo jobInfo)

Specified by:

createTranslationContext in interface SparkPortablePipelineTranslator<SparkStreamingTranslationContext>