Package org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions

Class SparkSideInputReader

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SparkSideInputReader

All Implemented Interfaces:

Serializable, org.apache.beam.runners.core.SideInputReader

public class SparkSideInputReader
extends Object
implements org.apache.beam.runners.core.SideInputReader, Serializable

SideInputReader using broadcasted SideInputValues.

See Also:

• Serialized Form

Method Summary

Modifier and Type	Method	Description
<T> boolean	contains(PCollectionView<T> view)
static org.apache.beam.runners.core.SideInputReader	create(Map<String,org.apache.spark.broadcast.Broadcast<SideInputValues<?>>> sideInputs)	Creates a SideInputReader for Spark from a map of PCollectionView tag ids and the corresponding broadcasted SideInputValues.
static org.apache.beam.runners.core.SideInputReader	empty()
<T> @Nullable T	get(PCollectionView<T> view, BoundedWindow window)
boolean	isEmpty()
static void	validateMaterializations(Iterable<PCollectionView<?>> views)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

empty

public static org.apache.beam.runners.core.SideInputReader empty()

create

public static org.apache.beam.runners.core.SideInputReader create(Map<String,org.apache.spark.broadcast.Broadcast<SideInputValues<?>>> sideInputs)

Creates a SideInputReader for Spark from a map of PCollectionView tag ids and the corresponding broadcasted SideInputValues.

Note, the materialization of respective PCollectionViews should be validated ahead of time before any costly creation and broadcast of SideInputValues.

validateMaterializations

public static void validateMaterializations(Iterable<PCollectionView<?>> views)

get

public <T> @Nullable T get(PCollectionView<T> view,
 BoundedWindow window)

Specified by:

get in interface org.apache.beam.runners.core.SideInputReader

contains

public <T> boolean contains(PCollectionView<T> view)

Specified by:

contains in interface org.apache.beam.runners.core.SideInputReader

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface org.apache.beam.runners.core.SideInputReader