Package org.apache.beam.runners.spark.util

Class GlobalWatermarkHolder

java.lang.Object

org.apache.beam.runners.spark.util.GlobalWatermarkHolder

public class GlobalWatermarkHolder
extends Object

A store to hold the global watermarks for a micro-batch.

For each source, holds a queue for the watermarks of each micro-batch that was read, and advances the watermarks according to the queue (first-in-first-out).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	GlobalWatermarkHolder.SparkWatermarks	A GlobalWatermarkHolder.SparkWatermarks holds the watermarks and batch time relevant to a micro-batch input from a specific source.
static class	GlobalWatermarkHolder.WatermarkAdvancingStreamingListener	Advance the WMs onBatchCompleted event.

Constructor Summary

Constructors

Constructor	Description
GlobalWatermarkHolder()

Method Summary

Modifier and Type	Method	Description
static void	add(int sourceId, GlobalWatermarkHolder.SparkWatermarks sparkWatermarks)
static void	addAll(Map<Integer,Queue<GlobalWatermarkHolder.SparkWatermarks>> sourceTimes)
static void	advance()	See advance(String).
static void	clear()
static Map<Integer,GlobalWatermarkHolder.SparkWatermarks>	get(Long cacheInterval)	Returns the Broadcast containing the GlobalWatermarkHolder.SparkWatermarks mapped to their sources.
static long	getLastWatermarkedBatchTime()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

GlobalWatermarkHolder

public GlobalWatermarkHolder()

Method Details

add

public static void add(int sourceId,
 GlobalWatermarkHolder.SparkWatermarks sparkWatermarks)

addAll

public static void addAll(Map<Integer,Queue<GlobalWatermarkHolder.SparkWatermarks>> sourceTimes)

getLastWatermarkedBatchTime

public static long getLastWatermarkedBatchTime()

get

public static Map<Integer,GlobalWatermarkHolder.SparkWatermarks> get(Long cacheInterval)

Returns the Broadcast containing the GlobalWatermarkHolder.SparkWatermarks mapped to their sources.

advance

public static void advance()

See advance(String).

clear

public static void clear()