Package org.apache.beam.runners.spark.translation

Interface SparkPortablePipelineTranslator<T extends SparkTranslationContext>

All Known Implementing Classes:

SparkBatchPortablePipelineTranslator, SparkStreamingPortablePipelineTranslator

public interface SparkPortablePipelineTranslator<T extends SparkTranslationContext>

Interface for portable Spark translators. This allows for a uniform invocation pattern for pipeline translation between streaming and batch runners.

Method Summary

Modifier and Type	Method	Description
T	createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, JobInfo jobInfo)
Set<String>	knownUrns()
void	translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline, T context)	Translates the given pipeline.

Method Details

knownUrns

Set<String> knownUrns()

translate

void translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline,
 T context)

Translates the given pipeline.

createTranslationContext

T createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc,
 SparkPipelineOptions options,
 JobInfo jobInfo)