Class SparkDatasetPortablePipelineTranslator
java.lang.Object
org.apache.beam.runners.spark.translation.SparkDatasetPortablePipelineTranslator
- All Implemented Interfaces:
SparkPortablePipelineTranslator<SparkDatasetTranslationContext>
public final class SparkDatasetPortablePipelineTranslator
extends Object
implements SparkPortablePipelineTranslator<SparkDatasetTranslationContext>
Translates a portable pipeline into Spark Dataset operations.
Executable stages run through the Fn API bridge of SparkExecutableStageFunction inside
mapPartitions. Side inputs are collected and broadcast. Unbounded input, user state and
timers are not supported yet and fail at translation.
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptioncreateTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, JobInfo jobInfo) voidtranslate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline, SparkDatasetTranslationContext context) Translates the given pipeline.
-
Constructor Details
-
SparkDatasetPortablePipelineTranslator
public SparkDatasetPortablePipelineTranslator()
-
-
Method Details
-
knownUrns
- Specified by:
knownUrnsin interfaceSparkPortablePipelineTranslator<SparkDatasetTranslationContext>
-
translate
public void translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline, SparkDatasetTranslationContext context) Description copied from interface:SparkPortablePipelineTranslatorTranslates the given pipeline.- Specified by:
translatein interfaceSparkPortablePipelineTranslator<SparkDatasetTranslationContext>
-
createTranslationContext
public SparkDatasetTranslationContext createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, JobInfo jobInfo) - Specified by:
createTranslationContextin interfaceSparkPortablePipelineTranslator<SparkDatasetTranslationContext>
-