org.apache.beam.runners.spark.structuredstreaming.translation

Class TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<? extends InT,OutT>>

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator<InT,OutT,TransformT>

Direct Known Subclasses:

ImpulseTranslatorBatch, ReshuffleTranslatorBatch, ReshuffleTranslatorBatch.ViaRandomKey

public abstract class TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<? extends InT,OutT>>
extends java.lang.Object

Supports translation between a Beam transform, and Spark's operations on Datasets.

WARNING: Do not make this class serializable! It could easily hide situations where unnecessary references leak into Spark closures.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	TransformTranslator.Context

Constructor Summary

Constructors

Constructor and Description
TransformTranslator()

Method Summary

Modifier and Type	Method and Description
void	translate(TransformT transform, org.apache.beam.sdk.runners.AppliedPTransform<InT,OutT,PTransform<InT,OutT>> appliedTransform, TranslationContext cxt)
protected abstract void	translate(TransformT transform, TransformTranslator.Context cxt)
protected <T> Coder<BoundedWindow>	windowCoder(PCollection<T> pc)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TransformTranslator

public TransformTranslator()

Method Detail

translate

protected abstract void translate(TransformT transform,
                                  TransformTranslator.Context cxt)
                           throws java.io.IOException

Throws:

java.io.IOException

translate

public final void translate(TransformT transform,
                            org.apache.beam.sdk.runners.AppliedPTransform<InT,OutT,PTransform<InT,OutT>> appliedTransform,
                            TranslationContext cxt)
                     throws java.io.IOException

Throws:

java.io.IOException

windowCoder

protected <T> Coder<BoundedWindow> windowCoder(PCollection<T> pc)