Package org.apache.beam.runners.spark.structuredstreaming.translation

Class TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<InT,OutT>>

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator<InT,OutT,TransformT>

@Internal
public abstract class TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<InT,OutT>>
extends Object

A TransformTranslator provides the capability to translate a specific primitive or composite PTransform into its Spark correspondence.

WARNING: TransformTranslators should never be serializable! This could easily hide situations where unnecessary references leak into Spark closures.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	TransformTranslator.Context	Available mutable context to translate a PTransform.

Field Summary

Fields

Modifier and Type	Field	Description
protected final float	complexityFactor

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	TransformTranslator(float complexityFactor)

Method Summary

Modifier and Type	Method	Description
protected boolean	canTranslate(TransformT transform)	Checks if a composite / primitive transform can be translated.
protected abstract void	translate(TransformT transform, TransformTranslator<InT,OutT,TransformT>.Context cxt)
protected <T> Coder<BoundedWindow>	windowCoder(PCollection<T> pc)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

complexityFactor

protected final float complexityFactor

Constructor Details

TransformTranslator

protected TransformTranslator(float complexityFactor)

Method Details

translate

protected abstract void translate(TransformT transform,
 TransformTranslator<InT,OutT,TransformT>.Context cxt)
                           throws IOException

Throws:

IOException

canTranslate

protected boolean canTranslate(TransformT transform)

Checks if a composite / primitive transform can be translated. Composites that cannot be translated as is, will be exploded further for translation of their parts.

This returns true by default and should be overridden where necessary.

Throws:

RuntimeException - If a transform uses unsupported features, an exception shall be thrown to give early feedback before any part of the pipeline is run.

windowCoder

protected <T> Coder<BoundedWindow> windowCoder(PCollection<T> pc)