org.apache.beam.runners.flink

Class FlinkBatchPortablePipelineTranslator

java.lang.Object

org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator

All Implemented Interfaces:

FlinkPortablePipelineTranslator<FlinkBatchPortablePipelineTranslator.BatchTranslationContext>

public class FlinkBatchPortablePipelineTranslator
extends java.lang.Object
implements FlinkPortablePipelineTranslator<FlinkBatchPortablePipelineTranslator.BatchTranslationContext>

A translator that translates bounded portable pipelines into executable Flink pipelines.

Example usage:

   FlinkBatchPortablePipelineTranslator translator =
       FlinkBatchPortablePipelineTranslator.createTranslator();
   BatchTranslationContext context =
       FlinkBatchPortablePipelineTranslator.createTranslationContext(jobInfo, confDir, filesToStage);
   translator.translate(context, pipeline);
   ExecutionEnvironment executionEnvironment = context.getExecutionEnvironment();
   // Do something with executionEnvironment...
 

After translation the ExecutionEnvironment in the translation context will contain the full not-yet-executed pipeline DAG corresponding to the input pipeline.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FlinkBatchPortablePipelineTranslator.BatchTranslationContext Batch translation context.
static class	FlinkBatchPortablePipelineTranslator.IsFlinkNativeTransform Predicate to determine whether a URN is a Flink native transform.
static interface	FlinkBatchPortablePipelineTranslator.PTransformTranslator Transform translation interface.

Nested classes/interfaces inherited from interface org.apache.beam.runners.flink.FlinkPortablePipelineTranslator

FlinkPortablePipelineTranslator.Executor, FlinkPortablePipelineTranslator.TranslationContext

Constructor Summary

Constructors

Constructor and Description
FlinkBatchPortablePipelineTranslator(java.util.Map<java.lang.String,FlinkBatchPortablePipelineTranslator.PTransformTranslator> urnToTransformTranslator)

Method Summary

Modifier and Type	Method and Description
static FlinkBatchPortablePipelineTranslator.BatchTranslationContext	createTranslationContext(JobInfo jobInfo, FlinkPipelineOptions pipelineOptions, org.apache.flink.api.java.ExecutionEnvironment executionEnvironment)
FlinkBatchPortablePipelineTranslator.BatchTranslationContext	createTranslationContext(JobInfo jobInfo, FlinkPipelineOptions pipelineOptions, @Nullable java.lang.String confDir, java.util.List<java.lang.String> filesToStage) Creates a batch translation context.
static FlinkBatchPortablePipelineTranslator	createTranslator() Creates a batch translator.
static FlinkBatchPortablePipelineTranslator	createTranslator(java.util.Map<java.lang.String,FlinkBatchPortablePipelineTranslator.PTransformTranslator> extraTranslations) Creates a batch translator.
java.util.Set<java.lang.String>	knownUrns()
FlinkPortablePipelineTranslator.Executor	translate(FlinkBatchPortablePipelineTranslator.BatchTranslationContext context, org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline) Translates the given pipeline.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.beam.runners.flink.FlinkPortablePipelineTranslator

prepareForTranslation

Constructor Detail

FlinkBatchPortablePipelineTranslator

public FlinkBatchPortablePipelineTranslator(java.util.Map<java.lang.String,FlinkBatchPortablePipelineTranslator.PTransformTranslator> urnToTransformTranslator)

Method Detail

createTranslationContext

public FlinkBatchPortablePipelineTranslator.BatchTranslationContext createTranslationContext(JobInfo jobInfo,
                                                                                             FlinkPipelineOptions pipelineOptions,
                                                                                             @Nullable java.lang.String confDir,
                                                                                             java.util.List<java.lang.String> filesToStage)

Creates a batch translation context. The resulting Flink execution dag will live in a new ExecutionEnvironment.

Specified by:

createTranslationContext in interface FlinkPortablePipelineTranslator<FlinkBatchPortablePipelineTranslator.BatchTranslationContext>

createTranslationContext

public static FlinkBatchPortablePipelineTranslator.BatchTranslationContext createTranslationContext(JobInfo jobInfo,
                                                                                                    FlinkPipelineOptions pipelineOptions,
                                                                                                    org.apache.flink.api.java.ExecutionEnvironment executionEnvironment)

createTranslator

public static FlinkBatchPortablePipelineTranslator createTranslator()

Creates a batch translator.

createTranslator

public static FlinkBatchPortablePipelineTranslator createTranslator(java.util.Map<java.lang.String,FlinkBatchPortablePipelineTranslator.PTransformTranslator> extraTranslations)

Creates a batch translator.

knownUrns

public java.util.Set<java.lang.String> knownUrns()

Specified by:

knownUrns in interface FlinkPortablePipelineTranslator<FlinkBatchPortablePipelineTranslator.BatchTranslationContext>

translate

public FlinkPortablePipelineTranslator.Executor translate(FlinkBatchPortablePipelineTranslator.BatchTranslationContext context,
                                                          org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline)

Description copied from interface: FlinkPortablePipelineTranslator

Translates the given pipeline.

Specified by:

translate in interface FlinkPortablePipelineTranslator<FlinkBatchPortablePipelineTranslator.BatchTranslationContext>