org.apache.beam.runners.spark.util

Class GlobalWatermarkHolder

java.lang.Object

org.apache.beam.runners.spark.util.GlobalWatermarkHolder

public class GlobalWatermarkHolder
extends java.lang.Object

A Broadcast variable to hold the global watermarks for a micro-batch.

For each source, holds a queue for the watermarks of each micro-batch that was read, and advances the watermarks according to the queue (first-in-first-out).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GlobalWatermarkHolder.SparkWatermarks A GlobalWatermarkHolder.SparkWatermarks holds the watermarks and batch time relevant to a micro-batch input from a specific source.
static class	GlobalWatermarkHolder.WatermarksListener Advance the WMs onBatchCompleted event.

Constructor Summary

Constructors

Constructor and Description
GlobalWatermarkHolder()

Method Summary

Modifier and Type	Method and Description
static void	add(int sourceId, GlobalWatermarkHolder.SparkWatermarks sparkWatermarks)
static void	addAll(java.util.Map<java.lang.Integer,java.util.Queue<GlobalWatermarkHolder.SparkWatermarks>> sourceTimes)
static void	advance(org.apache.spark.api.java.JavaSparkContext jsc) Advances the watermarks to the next-in-line watermarks.
static void	clear()
static org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks>>	get() Returns the Broadcast containing the GlobalWatermarkHolder.SparkWatermarks mapped to their sources.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GlobalWatermarkHolder

public GlobalWatermarkHolder()

Method Detail

add

public static void add(int sourceId,
                       GlobalWatermarkHolder.SparkWatermarks sparkWatermarks)

addAll

public static void addAll(java.util.Map<java.lang.Integer,java.util.Queue<GlobalWatermarkHolder.SparkWatermarks>> sourceTimes)

get

public static org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks>> get()

Returns the Broadcast containing the GlobalWatermarkHolder.SparkWatermarks mapped to their sources.

advance

public static void advance(org.apache.spark.api.java.JavaSparkContext jsc)

Advances the watermarks to the next-in-line watermarks. SparkWatermarks are monotonically increasing.

clear

public static void clear()