Package org.apache.beam.runners.spark.structuredstreaming.translation

Class PipelineTranslatorStreaming

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator

org.apache.beam.runners.spark.structuredstreaming.translation.batch.PipelineTranslatorCommon

org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslatorStreaming

@Internal
public class PipelineTranslatorStreaming
extends PipelineTranslatorCommon

Pipeline translator for streaming pipelines on Spark 4. It extends the common registry to reuse the stateless single output ParDo, Window.Assign, Flatten and Reshuffle translators, which are safe on a streaming Dataset. Every other primitive fails at translation, the batch translators for them persist or collect the Dataset, which Spark rejects for streaming plans.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator

PipelineTranslator.TranslationState, PipelineTranslator.UnresolvedTranslation<InT,T>

Constructor Summary

Constructors

Constructor	Description
PipelineTranslatorStreaming()

Method Summary

Modifier and Type	Method	Description
protected EvaluationContext	createEvaluationContext(Collection<? extends EvaluationContext.NamedDataset<?>> leaves, org.apache.spark.sql.SparkSession session, SparkCommonPipelineOptions options)	Creates the EvaluationContext for the translated pipeline.
protected <InT extends PInput, OutT extends POutput, TransformT extends PTransform<InT, OutT>> @Nullable TransformTranslator<InT,OutT,TransformT>	getTransformTranslator(TransformT transform)	Returns a TransformTranslator for the given PTransform if known.

Methods inherited from class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator

detectStreamingMode, replaceTransforms, translate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

PipelineTranslatorStreaming

public PipelineTranslatorStreaming()

Method Details

getTransformTranslator

protected <InT extends PInput,
OutT extends POutput,
TransformT extends PTransform<InT,
OutT>>
@Nullable TransformTranslator<InT,OutT,TransformT> getTransformTranslator(TransformT transform)

Returns a TransformTranslator for the given PTransform if known.

Overrides:

getTransformTranslator in class PipelineTranslatorCommon

createEvaluationContext

protected EvaluationContext createEvaluationContext(Collection<? extends EvaluationContext.NamedDataset<?>> leaves,
 org.apache.spark.sql.SparkSession session,
 SparkCommonPipelineOptions options)

Description copied from class: PipelineTranslator

Creates the EvaluationContext for the translated pipeline.

Subclasses may override this to return a specialized context, e.g. to evaluate streaming pipelines.

Overrides:

createEvaluationContext in class PipelineTranslator