Class SparkDatasetPortablePipelineTranslator

java.lang.Object
org.apache.beam.runners.spark.translation.SparkDatasetPortablePipelineTranslator
All Implemented Interfaces:
SparkPortablePipelineTranslator<SparkDatasetTranslationContext>

public final class SparkDatasetPortablePipelineTranslator extends Object implements SparkPortablePipelineTranslator<SparkDatasetTranslationContext>
Translates a portable pipeline into Spark Dataset operations.

Executable stages run through the Fn API bridge of SparkExecutableStageFunction inside mapPartitions. Side inputs are collected and broadcast. Unbounded input, user state and timers are not supported yet and fail at translation.