org.apache.beam.runners.spark.structuredstreaming.translation.batch

Class CombineGroupedValuesTranslatorBatch<K,InT,AccT,OutT>

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator<PCollection<? extends KV<K,? extends java.lang.Iterable<InT>>>,PCollection<KV<K,OutT>>,Combine.GroupedValues<K,InT,OutT>>

org.apache.beam.runners.spark.structuredstreaming.translation.batch.CombineGroupedValuesTranslatorBatch<K,InT,AccT,OutT>

public class CombineGroupedValuesTranslatorBatch<K,InT,AccT,OutT>
extends TransformTranslator<PCollection<? extends KV<K,? extends java.lang.Iterable<InT>>>,PCollection<KV<K,OutT>>,Combine.GroupedValues<K,InT,OutT>>

Translator for Combine.GroupedValues if the Combine.CombineFn doesn't require context / side-inputs.

This doesn't require a Spark Aggregator. Instead it can directly use the respective Combine.CombineFn to reduce each iterable of values into an aggregated output value.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator

TransformTranslator.Context

Constructor Summary

Constructors

Constructor and Description
CombineGroupedValuesTranslatorBatch()

Method Summary

Modifier and Type	Method and Description
boolean	canTranslate(Combine.GroupedValues<K,InT,OutT> transform) Checks if a composite / primitive transform can be translated.
protected void	translate(Combine.GroupedValues<K,InT,OutT> transform, TransformTranslator.Context cxt)

Methods inherited from class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator

windowCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CombineGroupedValuesTranslatorBatch

public CombineGroupedValuesTranslatorBatch()

Method Detail

translate

protected void translate(Combine.GroupedValues<K,InT,OutT> transform,
                         TransformTranslator.Context cxt)
                  throws java.io.IOException

Specified by:

translate in class TransformTranslator<PCollection<? extends KV<K,? extends java.lang.Iterable<InT>>>,PCollection<KV<K,OutT>>,Combine.GroupedValues<K,InT,OutT>>

Throws:

java.io.IOException

canTranslate

public boolean canTranslate(Combine.GroupedValues<K,InT,OutT> transform)

Description copied from class: TransformTranslator

Checks if a composite / primitive transform can be translated. Composites that cannot be translated as is, will be exploded further for translation of their parts.

This should be overridden where necessary. If a transform is know to be unsupported, this should throw a runtime exception to give early feedback before any part of the pipeline is run.

Overrides:

canTranslate in class TransformTranslator<PCollection<? extends KV<K,? extends java.lang.Iterable<InT>>>,PCollection<KV<K,OutT>>,Combine.GroupedValues<K,InT,OutT>>