Package org.apache.beam.runners.spark.translation

Class SparkBatchPortablePipelineTranslator

java.lang.Object

org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator

All Implemented Interfaces:

SparkPortablePipelineTranslator<SparkTranslationContext>

public class SparkBatchPortablePipelineTranslator
extends Object
implements SparkPortablePipelineTranslator<SparkTranslationContext>

Translates a bounded portable pipeline into a Spark job.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	SparkBatchPortablePipelineTranslator.IsSparkNativeTransform	Predicate to determine whether a URN is a Spark native transform.

Constructor Summary

Constructors

Constructor	Description
SparkBatchPortablePipelineTranslator()

Method Summary

Modifier and Type	Method	Description
SparkTranslationContext	createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, JobInfo jobInfo)
Set<String>	knownUrns()
void	translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline, SparkTranslationContext context)	Translates pipeline from Beam into the Spark context.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SparkBatchPortablePipelineTranslator

public SparkBatchPortablePipelineTranslator()

Method Details

knownUrns

public Set<String> knownUrns()

Specified by:

knownUrns in interface SparkPortablePipelineTranslator<SparkTranslationContext>

translate

public void translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline,
 SparkTranslationContext context)

Translates pipeline from Beam into the Spark context.

Specified by:

translate in interface SparkPortablePipelineTranslator<SparkTranslationContext>

createTranslationContext

public SparkTranslationContext createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc,
 SparkPipelineOptions options,
 JobInfo jobInfo)

Specified by:

createTranslationContext in interface SparkPortablePipelineTranslator<SparkTranslationContext>