org.apache.beam.runners.spark.structuredstreaming.translation

Class PipelineTranslator

java.lang.Object

org.apache.beam.sdk.Pipeline.PipelineVisitor.Defaults

org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator

All Implemented Interfaces:

Pipeline.PipelineVisitor

Direct Known Subclasses:

PipelineTranslatorBatch, PipelineTranslatorStreaming

public abstract class PipelineTranslator
extends Pipeline.PipelineVisitor.Defaults

Pipeline.PipelineVisitor that translates the Beam operators to their Spark counterparts. It also does the pipeline preparation: mode detection, transforms replacement, classpath preparation. If we have a streaming job, it is instantiated as a PipelineTranslatorStreaming. If we have a batch job, it is instantiated as a PipelineTranslatorBatch.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.Pipeline.PipelineVisitor

Pipeline.PipelineVisitor.CompositeBehavior, Pipeline.PipelineVisitor.Defaults

Field Summary

Fields

Modifier and Type	Field and Description
protected TranslationContext	translationContext

Constructor Summary

Constructors

Constructor and Description
PipelineTranslator()

Method Summary

Modifier and Type	Method and Description
static void	detectTranslationMode(Pipeline pipeline, StreamingOptions options) Visit the pipeline to determine the translation mode (batch/streaming) and update options accordingly.
Pipeline.PipelineVisitor.CompositeBehavior	enterCompositeTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Called for each composite transform after all topological predecessors have been visited but before any of its component transforms.
protected abstract TransformTranslator<?>	getTransformTranslator(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Get a TransformTranslator for the given TransformHierarchy.Node.
TranslationContext	getTranslationContext()
static boolean	isLocalSparkMaster(SparkStructuredStreamingPipelineOptions options) Detects if the pipeline is run in local spark mode.
void	leaveCompositeTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Called for each composite transform after all of its component transforms and their outputs have been visited.
static void	prepareFilesToStageForRemoteClusterExecution(SparkStructuredStreamingPipelineOptions options) Local configurations work in the same JVM and have no problems with improperly formatted files on classpath (eg.
static void	replaceTransforms(Pipeline pipeline, StreamingOptions options)
void	translate(Pipeline pipeline) Translates the pipeline by passing this class as a visitor.
void	visitPrimitiveTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Called for each primitive transform after all of its topological predecessors and inputs have been visited.

Methods inherited from class org.apache.beam.sdk.Pipeline.PipelineVisitor.Defaults

enterPipeline, getPipeline, leavePipeline, visitValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

translationContext

protected TranslationContext translationContext

Constructor Detail

PipelineTranslator

public PipelineTranslator()

Method Detail

prepareFilesToStageForRemoteClusterExecution

public static void prepareFilesToStageForRemoteClusterExecution(SparkStructuredStreamingPipelineOptions options)

Local configurations work in the same JVM and have no problems with improperly formatted files on classpath (eg. directories with .class files or empty directories). Prepare files for staging only when using remote cluster (passing the master address explicitly).

replaceTransforms

public static void replaceTransforms(Pipeline pipeline,
                                     StreamingOptions options)

detectTranslationMode

public static void detectTranslationMode(Pipeline pipeline,
                                         StreamingOptions options)

Visit the pipeline to determine the translation mode (batch/streaming) and update options accordingly.

isLocalSparkMaster

public static boolean isLocalSparkMaster(SparkStructuredStreamingPipelineOptions options)

Detects if the pipeline is run in local spark mode.

getTransformTranslator

protected abstract TransformTranslator<?> getTransformTranslator(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Get a TransformTranslator for the given TransformHierarchy.Node.

translate

public void translate(Pipeline pipeline)

Translates the pipeline by passing this class as a visitor.

Parameters:

pipeline - The pipeline to be translated

enterCompositeTransform

public Pipeline.PipelineVisitor.CompositeBehavior enterCompositeTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Description copied from interface: Pipeline.PipelineVisitor

Called for each composite transform after all topological predecessors have been visited but before any of its component transforms.

The return value controls whether or not child transforms are visited.

Specified by:

enterCompositeTransform in interface Pipeline.PipelineVisitor

Overrides:

enterCompositeTransform in class Pipeline.PipelineVisitor.Defaults

leaveCompositeTransform

public void leaveCompositeTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Description copied from interface: Pipeline.PipelineVisitor

Called for each composite transform after all of its component transforms and their outputs have been visited.

Specified by:

leaveCompositeTransform in interface Pipeline.PipelineVisitor

Overrides:

leaveCompositeTransform in class Pipeline.PipelineVisitor.Defaults

visitPrimitiveTransform

public void visitPrimitiveTransform(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Description copied from interface: Pipeline.PipelineVisitor

Called for each primitive transform after all of its topological predecessors and inputs have been visited.

Specified by:

visitPrimitiveTransform in interface Pipeline.PipelineVisitor

Overrides:

visitPrimitiveTransform in class Pipeline.PipelineVisitor.Defaults

getTranslationContext

public TranslationContext getTranslationContext()