org.apache.beam.runners.spark.structuredstreaming.translation.streaming

Class PipelineTranslatorStreaming

java.lang.Object

org.apache.beam.sdk.Pipeline.PipelineVisitor.Defaults

org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator

org.apache.beam.runners.spark.structuredstreaming.translation.streaming.PipelineTranslatorStreaming

All Implemented Interfaces:

Pipeline.PipelineVisitor

public class PipelineTranslatorStreaming
extends PipelineTranslator

PipelineTranslator for executing a Pipeline in Spark in streaming mode. This contains only the components specific to streaming: registry of streaming TransformTranslator and registry lookup code.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.Pipeline.PipelineVisitor

Pipeline.PipelineVisitor.CompositeBehavior, Pipeline.PipelineVisitor.Defaults

Field Summary

Fields inherited from class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator

translationContext

Constructor Summary

Constructors

Constructor and Description
PipelineTranslatorStreaming(SparkStructuredStreamingPipelineOptions options)

Method Summary

Modifier and Type	Method and Description
protected TransformTranslator<?>	getTransformTranslator(org.apache.beam.sdk.runners.TransformHierarchy.Node node) Returns a translator for the given node, if it is possible, otherwise null.

Methods inherited from class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator

detectTranslationMode, enterCompositeTransform, getTranslationContext, isLocalSparkMaster, leaveCompositeTransform, prepareFilesToStageForRemoteClusterExecution, replaceTransforms, translate, visitPrimitiveTransform

Methods inherited from class org.apache.beam.sdk.Pipeline.PipelineVisitor.Defaults

enterPipeline, getPipeline, leavePipeline, visitValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PipelineTranslatorStreaming

public PipelineTranslatorStreaming(SparkStructuredStreamingPipelineOptions options)

Method Detail

getTransformTranslator

protected TransformTranslator<?> getTransformTranslator(org.apache.beam.sdk.runners.TransformHierarchy.Node node)

Returns a translator for the given node, if it is possible, otherwise null.

Specified by:

getTransformTranslator in class PipelineTranslator