org.apache.beam.runners.spark.util

Class GlobalWatermarkHolder

java.lang.Object

org.apache.beam.runners.spark.util.GlobalWatermarkHolder

public class GlobalWatermarkHolder
extends java.lang.Object

A store to hold the global watermarks for a micro-batch.

For each source, holds a queue for the watermarks of each micro-batch that was read, and advances the watermarks according to the queue (first-in-first-out).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GlobalWatermarkHolder.SparkWatermarks A GlobalWatermarkHolder.SparkWatermarks holds the watermarks and batch time relevant to a micro-batch input from a specific source.
static class	GlobalWatermarkHolder.WatermarkAdvancingStreamingListener Advance the WMs onBatchCompleted event.

Constructor Summary

Constructors

Constructor and Description
GlobalWatermarkHolder()

Method Summary

Modifier and Type	Method and Description
static void	add(int sourceId, GlobalWatermarkHolder.SparkWatermarks sparkWatermarks)
static void	addAll(java.util.Map<java.lang.Integer,java.util.Queue<GlobalWatermarkHolder.SparkWatermarks>> sourceTimes)
static void	advance() See advance(String).
static void	advance(java.lang.String batchId) Advances the watermarks to the next-in-line watermarks.
static void	clear()
static java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks>	get(java.lang.Long cacheInterval) Returns the Broadcast containing the GlobalWatermarkHolder.SparkWatermarks mapped to their sources.
static long	getLastWatermarkedBatchTime()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GlobalWatermarkHolder

public GlobalWatermarkHolder()

Method Detail

add

public static void add(int sourceId,
                       GlobalWatermarkHolder.SparkWatermarks sparkWatermarks)

addAll

public static void addAll(java.util.Map<java.lang.Integer,java.util.Queue<GlobalWatermarkHolder.SparkWatermarks>> sourceTimes)

getLastWatermarkedBatchTime

public static long getLastWatermarkedBatchTime()

get

public static java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks> get(java.lang.Long cacheInterval)

Returns the Broadcast containing the GlobalWatermarkHolder.SparkWatermarks mapped to their sources.

advance

public static void advance(java.lang.String batchId)

Advances the watermarks to the next-in-line watermarks. SparkWatermarks are monotonically increasing.

advance

public static void advance()

See advance(String).

clear

public static void clear()