org.apache.beam.runners.spark.stateful

Class SparkTimerInternals

java.lang.Object

org.apache.beam.runners.spark.stateful.SparkTimerInternals

All Implemented Interfaces:

org.apache.beam.runners.core.TimerInternals

public class SparkTimerInternals
extends java.lang.Object
implements org.apache.beam.runners.core.TimerInternals

An implementation of TimerInternals for the SparkRunner.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.runners.core.TimerInternals

org.apache.beam.runners.core.TimerInternals.TimerData, org.apache.beam.runners.core.TimerInternals.TimerDataCoder

Method Summary

Modifier and Type	Method and Description
void	advanceWatermark() Advances the watermark.
Instant	currentInputWatermarkTime() Return the current, local input watermark timestamp for this computation in the TimeDomain.EVENT_TIME time domain.
Instant	currentOutputWatermarkTime() Return the current, local output watermark timestamp for this computation in the TimeDomain.EVENT_TIME time domain.
Instant	currentProcessingTime() Returns the current timestamp in the TimeDomain.PROCESSING_TIME time domain.
Instant	currentSynchronizedProcessingTime() Returns the current timestamp in the TimeDomain.SYNCHRONIZED_PROCESSING_TIME time domain or null if unknown.
void	deleteTimer(org.apache.beam.runners.core.StateNamespace namespace, java.lang.String timerId)
void	deleteTimer(org.apache.beam.runners.core.StateNamespace namespace, java.lang.String timerId, TimeDomain timeDomain) Deletes the given timer.
void	deleteTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)
static java.lang.Iterable<org.apache.beam.runners.core.TimerInternals.TimerData>	deserializeTimers(java.util.Collection<byte[]> serTimers, org.apache.beam.runners.core.TimerInternals.TimerDataCoder timerDataCoder)
static SparkTimerInternals	forStreamFromSources(java.util.List<java.lang.Integer> sourceIds, org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks>> broadcast) Build the TimerInternals according to the feeding streams.
static SparkTimerInternals	global(org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks>> broadcast) Build a global TimerInternals for all feeding streams.
static java.util.Collection<byte[]>	serializeTimers(java.util.Collection<org.apache.beam.runners.core.TimerInternals.TimerData> timers, org.apache.beam.runners.core.TimerInternals.TimerDataCoder timerDataCoder)
void	setTimer(org.apache.beam.runners.core.StateNamespace namespace, java.lang.String timerId, Instant target, TimeDomain timeDomain) Sets a timer to be fired when the current time in the specified time domain reaches the target timestamp.
void	setTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

forStreamFromSources

public static SparkTimerInternals forStreamFromSources(java.util.List<java.lang.Integer> sourceIds,
                                                       @Nullable
                                                       org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks>> broadcast)

Build the TimerInternals according to the feeding streams.

global

public static SparkTimerInternals global(@Nullable
                                         org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks>> broadcast)

Build a global TimerInternals for all feeding streams.

setTimer

public void setTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)

Specified by:

setTimer in interface org.apache.beam.runners.core.TimerInternals

deleteTimer

public void deleteTimer(org.apache.beam.runners.core.StateNamespace namespace,
                        java.lang.String timerId,
                        TimeDomain timeDomain)

Description copied from interface: org.apache.beam.runners.core.TimerInternals

Deletes the given timer.

A timer's ID is enforced to be unique in validation of a DoFn, but runners often manage timers for different time domains in very different ways, thus the TimeDomain is a required parameter.

Specified by:

deleteTimer in interface org.apache.beam.runners.core.TimerInternals

deleteTimer

public void deleteTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)

Specified by:

deleteTimer in interface org.apache.beam.runners.core.TimerInternals

currentProcessingTime

public Instant currentProcessingTime()

Description copied from interface: org.apache.beam.runners.core.TimerInternals

Returns the current timestamp in the TimeDomain.PROCESSING_TIME time domain.

Specified by:

currentProcessingTime in interface org.apache.beam.runners.core.TimerInternals

currentSynchronizedProcessingTime

@Nullable
public Instant currentSynchronizedProcessingTime()

Description copied from interface: org.apache.beam.runners.core.TimerInternals

Returns the current timestamp in the TimeDomain.SYNCHRONIZED_PROCESSING_TIME time domain or null if unknown.

Specified by:

currentSynchronizedProcessingTime in interface org.apache.beam.runners.core.TimerInternals

currentInputWatermarkTime

public Instant currentInputWatermarkTime()

Description copied from interface: org.apache.beam.runners.core.TimerInternals

Return the current, local input watermark timestamp for this computation in the TimeDomain.EVENT_TIME time domain.

This value:

1. Is never null, but may be BoundedWindow.TIMESTAMP_MIN_VALUE.
2. Is monotonically increasing.
3. May differ between workers due to network and other delays.
4. Will never be ahead of the global input watermark for this computation. But it may be arbitrarily behind the global input watermark.
5. Any element with a timestamp before the local input watermark can be considered 'locally late' and be subject to special processing or be dropped entirely.

Note that because the local input watermark can be behind the global input watermark, it is possible for an element to be considered locally on-time even though it is globally late.

Specified by:

currentInputWatermarkTime in interface org.apache.beam.runners.core.TimerInternals

advanceWatermark

public void advanceWatermark()

Advances the watermark.

currentOutputWatermarkTime

@Nullable
public Instant currentOutputWatermarkTime()

Description copied from interface: org.apache.beam.runners.core.TimerInternals

Return the current, local output watermark timestamp for this computation in the TimeDomain.EVENT_TIME time domain. Return null if unknown.

This value:

1. Is monotonically increasing.
2. Will never be ahead of TimerInternals.currentInputWatermarkTime() as returned above.
3. May differ between workers due to network and other delays.
4. However will never be behind the global input watermark for any following computation.

In pictures:

  |              |       |       |       |
  |              |   D   |   C   |   B   |   A
  |              |       |       |       |
 GIWM     <=    GOWM <= LOWM <= LIWM <= GIWM
 (next stage)
 -------------------------------------------------> event time
 

where

• LOWM = local output water mark.
• GOWM = global output water mark.
• GIWM = global input water mark.
• LIWM = local input water mark.
• A = A globally on-time element.
• B = A globally late, but locally on-time element.
• C = A locally late element which may still contribute to the timestamp of a pane.
• D = A locally late element which cannot contribute to the timestamp of a pane.

Note that if a computation emits an element which is not before the current output watermark then that element will always appear locally on-time in all following computations. However, it is possible for an element emitted before the current output watermark to appear locally on-time in a following computation. Thus we must be careful to never assume locally late data viewed on the output of a computation remains locally late on the input of a following computation.

Specified by:

currentOutputWatermarkTime in interface org.apache.beam.runners.core.TimerInternals

setTimer

public void setTimer(org.apache.beam.runners.core.StateNamespace namespace,
                     java.lang.String timerId,
                     Instant target,
                     TimeDomain timeDomain)

Description copied from interface: org.apache.beam.runners.core.TimerInternals

Sets a timer to be fired when the current time in the specified time domain reaches the target timestamp.

The combination of namespace and timerId uniquely identify a timer.

If a timer is set and then set again before it fires, later settings should clear the prior setting.

It is an error to set a timer for two different time domains.

Specified by:

setTimer in interface org.apache.beam.runners.core.TimerInternals

deleteTimer

public void deleteTimer(org.apache.beam.runners.core.StateNamespace namespace,
                        java.lang.String timerId)

Specified by:

deleteTimer in interface org.apache.beam.runners.core.TimerInternals

serializeTimers

public static java.util.Collection<byte[]> serializeTimers(java.util.Collection<org.apache.beam.runners.core.TimerInternals.TimerData> timers,
                                                           org.apache.beam.runners.core.TimerInternals.TimerDataCoder timerDataCoder)

deserializeTimers

public static java.lang.Iterable<org.apache.beam.runners.core.TimerInternals.TimerData> deserializeTimers(java.util.Collection<byte[]> serTimers,
                                                                                                          org.apache.beam.runners.core.TimerInternals.TimerDataCoder timerDataCoder)