org.apache.beam.runners.spark.structuredstreaming.translation

Interface PipelineTranslator.TranslationState

All Superinterfaces:

EncoderProvider

All Known Implementing Classes:

TransformTranslator.Context

Enclosing class:

PipelineTranslator

public static interface PipelineTranslator.TranslationState
extends EncoderProvider

Shared, mutable state during the translation of a pipeline and omitted afterwards.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider

EncoderProvider.Factory<T>

Method Summary

Modifier and Type	Method and Description
<T> org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<T>>	getDataset(PCollection<T> pCollection)
PipelineOptions	getOptions()
java.util.function.Supplier<PipelineOptions>	getOptionsSupplier()
<T> org.apache.spark.broadcast.Broadcast<SideInputValues<T>>	getSideInputBroadcast(PCollection<T> pCollection, SideInputValues.Loader<T> loader)
org.apache.spark.sql.SparkSession	getSparkSession()
boolean	isLeave(PCollection<?> pCollection)
default <T> void	putDataset(PCollection<T> pCollection, org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<T>> dataset)
<T> void	putDataset(PCollection<T> pCollection, org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<T>> dataset, boolean cache)

Methods inherited from interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider

encoderFactory, encoderOf, encoderOf, keyEncoderOf, kvEncoderOf, valueEncoderOf

Method Detail

getDataset

<T> org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<T>> getDataset(PCollection<T> pCollection)

isLeave

boolean isLeave(PCollection<?> pCollection)

putDataset

<T> void putDataset(PCollection<T> pCollection,
                    org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<T>> dataset,
                    boolean cache)

putDataset

default <T> void putDataset(PCollection<T> pCollection,
                            org.apache.spark.sql.Dataset<org.apache.beam.sdk.util.WindowedValue<T>> dataset)

getSideInputBroadcast

<T> org.apache.spark.broadcast.Broadcast<SideInputValues<T>> getSideInputBroadcast(PCollection<T> pCollection,
                                                                                   SideInputValues.Loader<T> loader)

getOptionsSupplier

java.util.function.Supplier<PipelineOptions> getOptionsSupplier()

getOptions

PipelineOptions getOptions()

getSparkSession

org.apache.spark.sql.SparkSession getSparkSession()