Package org.apache.beam.runners.spark.structuredstreaming.translation

Interface PipelineTranslator.UnresolvedTranslation<InT,T>

Enclosing class:

PipelineTranslator

public static interface PipelineTranslator.UnresolvedTranslation<InT,T>

Unresolved translation, allowing to optimize the generated Spark DAG.

An unresolved translation can - in certain cases - be fused together with following transforms. Currently this is only the case for ParDos with linear linage.

Method Summary

Modifier and Type	Method	Description
<T2> PipelineTranslator.UnresolvedTranslation<InT,T2>	fuse(PipelineTranslator.UnresolvedTranslation<T,T2> next)
PCollection<InT>	getInput()
org.apache.spark.sql.Dataset<WindowedValue<T>>	resolve(Supplier<PipelineOptions> options, org.apache.spark.sql.Dataset<WindowedValue<InT>> input)

Method Details

getInput

PCollection<InT> getInput()

fuse

<T2>
PipelineTranslator.UnresolvedTranslation<InT,T2> fuse(PipelineTranslator.UnresolvedTranslation<T,T2> next)

resolve

org.apache.spark.sql.Dataset<WindowedValue<T>> resolve(Supplier<PipelineOptions> options,
 org.apache.spark.sql.Dataset<WindowedValue<InT>> input)