org.apache.beam.runners.spark.structuredstreaming.translation

Interface PipelineTranslator.UnresolvedTranslation<InT,T>

Enclosing class:

PipelineTranslator

public static interface PipelineTranslator.UnresolvedTranslation<InT,T>

Unresolved translation, allowing to optimize the generated Spark DAG.

An unresolved translation can - in certain cases - be fused together with following transforms. Currently this is only the case for ParDos with linear linage.

Method Summary

Modifier and Type	Method and Description
<T2> PipelineTranslator.UnresolvedTranslation<InT,T2>	fuse(PipelineTranslator.UnresolvedTranslation<T,T2> next)
PCollection<InT>	getInput()
org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<T>>	resolve(java.util.function.Supplier<PipelineOptions> options, org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<InT>> input)

Method Detail

getInput

PCollection<InT> getInput()

fuse

<T2> PipelineTranslator.UnresolvedTranslation<InT,T2> fuse(PipelineTranslator.UnresolvedTranslation<T,T2> next)

resolve

org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<T>> resolve(java.util.function.Supplier<PipelineOptions> options,
                                                                                org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<InT>> input)