org.apache.beam.runners.spark.structuredstreaming.translation

Class TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<? extends InT,OutT>>

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator<InT,OutT,TransformT>

Direct Known Subclasses:

CombineGroupedValuesTranslatorBatch, ImpulseTranslatorBatch, ReshuffleTranslatorBatch, ReshuffleTranslatorBatch.ViaRandomKey

@Internal
public abstract class TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<? extends InT,OutT>>
extends java.lang.Object

A TransformTranslator provides the capability to translate a specific primitive or composite PTransform into its Spark correspondence.

WARNING: TransformTranslators should never be serializable! This could easily hide situations where unnecessary references leak into Spark closures.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	TransformTranslator.Context Available mutable context to translate a PTransform.

Constructor Summary

Constructors

Constructor and Description
TransformTranslator()

Method Summary

Modifier and Type	Method and Description
protected boolean	canTranslate(TransformT transform) Checks if a composite / primitive transform can be translated.
protected abstract void	translate(TransformT transform, TransformTranslator.Context cxt)
protected <T> Coder<BoundedWindow>	windowCoder(PCollection<T> pc)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TransformTranslator

public TransformTranslator()

Method Detail

translate

protected abstract void translate(TransformT transform,
                                  TransformTranslator.Context cxt)
                           throws java.io.IOException

Throws:

java.io.IOException

canTranslate

protected boolean canTranslate(TransformT transform)

Checks if a composite / primitive transform can be translated. Composites that cannot be translated as is, will be exploded further for translation of their parts.

This should be overridden where necessary. If a transform is know to be unsupported, this should throw a runtime exception to give early feedback before any part of the pipeline is run.

windowCoder

protected <T> Coder<BoundedWindow> windowCoder(PCollection<T> pc)