org.apache.beam.runners.spark.structuredstreaming.translation

Class PipelineTranslator

java.lang.Object

org.apache.beam.sdk.Pipeline.PipelineVisitor.Defaults

org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator

All Implemented Interfaces:

Pipeline.PipelineVisitor

Direct Known Subclasses:

PipelineTranslatorBatch, PipelineTranslatorStreaming

public abstract class PipelineTranslator
extends Pipeline.PipelineVisitor.Defaults

Pipeline.PipelineVisitor that translates the Beam operators to their Spark counterparts. It also does the pipeline preparation: mode detection, transforms replacement, classpath preparation. If we have a streaming job, it is instantiated as a PipelineTranslatorStreaming. If we have a batch job, it is instantiated as a PipelineTranslatorBatch.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.Pipeline.PipelineVisitor

Pipeline.PipelineVisitor.CompositeBehavior, Pipeline.PipelineVisitor.Defaults

Field Summary

Fields

Modifier and Type	Field and Description
protected AbstractTranslationContext	translationContext

Constructor Summary

Constructors

Constructor and Description
PipelineTranslator()

Method Summary

Modifier and Type	Method and Description
static void	detectTranslationMode(Pipeline pipeline, StreamingOptions options) Visit the pipeline to determine the translation mode (batch/streaming) and update options accordingly.
Pipeline.PipelineVisitor.CompositeBehavior	enterCompositeTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Called for each composite transform after all topological predecessors have been visited but before any of its component transforms.
protected abstract TransformTranslator<?>	getTransformTranslator(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Get a TransformTranslator for the given TransformHierarchy.Node.
AbstractTranslationContext	getTranslationContext()
void	leaveCompositeTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Called for each composite transform after all of its component transforms and their outputs have been visited.
static void	replaceTransforms(Pipeline pipeline, StreamingOptions options)
void	translate(Pipeline pipeline) Translates the pipeline by passing this class as a visitor.
void	visitPrimitiveTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Called for each primitive transform after all of its topological predecessors and inputs have been visited.

Methods inherited from class org.apache.beam.sdk.Pipeline.PipelineVisitor.Defaults

enterPipeline, getPipeline, leavePipeline, visitValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

translationContext

protected AbstractTranslationContext translationContext

Constructor Detail

PipelineTranslator

public PipelineTranslator()

Method Detail

replaceTransforms

public static void replaceTransforms(Pipeline pipeline,
                                     StreamingOptions options)

detectTranslationMode

public static void detectTranslationMode(Pipeline pipeline,
                                         StreamingOptions options)

Visit the pipeline to determine the translation mode (batch/streaming) and update options accordingly.

getTransformTranslator

protected abstract TransformTranslator<?> getTransformTranslator(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Get a TransformTranslator for the given TransformHierarchy.Node.

translate

public void translate(Pipeline pipeline)

Translates the pipeline by passing this class as a visitor.

Parameters:

pipeline - The pipeline to be translated

enterCompositeTransform

public Pipeline.PipelineVisitor.CompositeBehavior enterCompositeTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Description copied from interface: Pipeline.PipelineVisitor

Called for each composite transform after all topological predecessors have been visited but before any of its component transforms.

The return value controls whether or not child transforms are visited.

Specified by:

enterCompositeTransform in interface Pipeline.PipelineVisitor

Overrides:

enterCompositeTransform in class Pipeline.PipelineVisitor.Defaults

leaveCompositeTransform

public void leaveCompositeTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Description copied from interface: Pipeline.PipelineVisitor

Called for each composite transform after all of its component transforms and their outputs have been visited.

Specified by:

leaveCompositeTransform in interface Pipeline.PipelineVisitor

Overrides:

leaveCompositeTransform in class Pipeline.PipelineVisitor.Defaults

visitPrimitiveTransform

public void visitPrimitiveTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Description copied from interface: Pipeline.PipelineVisitor

Called for each primitive transform after all of its topological predecessors and inputs have been visited.

Specified by:

visitPrimitiveTransform in interface Pipeline.PipelineVisitor

Overrides:

visitPrimitiveTransform in class Pipeline.PipelineVisitor.Defaults

getTranslationContext

public AbstractTranslationContext getTranslationContext()