org.apache.beam.runners.spark.structuredstreaming.translation

Class TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<InT,OutT>>

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator<InT,OutT,TransformT>

@Internal
public abstract class TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<InT,OutT>>
extends java.lang.Object

A TransformTranslator provides the capability to translate a specific primitive or composite PTransform into its Spark correspondence.

WARNING: TransformTranslators should never be serializable! This could easily hide situations where unnecessary references leak into Spark closures.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	TransformTranslator.Context Available mutable context to translate a PTransform.

Field Summary

Fields

Modifier and Type	Field and Description
protected float	complexityFactor

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	TransformTranslator(float complexityFactor)

Method Summary

Modifier and Type	Method and Description
protected boolean	canTranslate(TransformT transform) Checks if a composite / primitive transform can be translated.
protected abstract void	translate(TransformT transform, TransformTranslator.Context cxt)
protected <T> Coder<BoundedWindow>	windowCoder(PCollection<T> pc)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

complexityFactor

protected final float complexityFactor

Constructor Detail

TransformTranslator

protected TransformTranslator(float complexityFactor)

Method Detail

translate

protected abstract void translate(TransformT transform,
                                  TransformTranslator.Context cxt)
                           throws java.io.IOException

Throws:

java.io.IOException

canTranslate

protected boolean canTranslate(TransformT transform)

Checks if a composite / primitive transform can be translated. Composites that cannot be translated as is, will be exploded further for translation of their parts.

This returns true by default and should be overridden where necessary.

Throws:

java.lang.RuntimeException - If a transform uses unsupported features, an exception shall be thrown to give early feedback before any part of the pipeline is run.

windowCoder

protected <T> Coder<BoundedWindow> windowCoder(PCollection<T> pc)