org.apache.beam.sdk.transforms

Class Wait

java.lang.Object

org.apache.beam.sdk.transforms.Wait

@Experimental
public class Wait
extends java.lang.Object

Delays processing of each window in a PCollection until signaled.

Given a main PCollection and a signal PCollection, produces output identical to its main input, but all elements for a window are produced only once that window is closed in the signal PCollection.

To express the pattern "apply T to X after Y is ready", use X.apply(Wait.on(Y)).apply(T).

In particular: returns a PCollection with contents identical to the input, but delays producing elements of the output in window W until the signal's window W closes (i.e. signal's watermark passes W.end + signal.allowedLateness).

In other words, an element of the output at timestamp "t" will be produced only after no more elements of the signal can appear with a timestamp below "t".

Example usage: write a PCollection to one database and then to another database, making sure that writing a window of data to the second database starts only after the respective window has been fully written to the first database.

 PCollection<Void> firstWriteResults = data.apply(ParDo.of(...write to first database...));
 data.apply(Wait.on(firstWriteResults))
     // Windows of this intermediate PCollection will be processed no earlier than when
     // the respective window of firstWriteResults closes.
     .apply(ParDo.of(...write to second database...));
 

Notes:

• If signal is globally windowed, main input must also be. This typically would be useful only in a batch pipeline, because the global window of an infinite PCollection never closes, so the wait signal will never be ready.
• Beware that if the signal has large allowed lateness, the wait signal will fire only after that lateness elapses, i.e. after the watermark of the signal passes end of the window plus allowed lateness. In other words: do not use this with signals that specify a large allowed lateness.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Wait.OnSignal<T> Implementation of on(org.apache.beam.sdk.values.PCollection<?>...).

Constructor Summary

Constructors

Constructor and Description
Wait()

Method Summary

Modifier and Type	Method and Description
static <T> Wait.OnSignal<T>	on(java.util.List<PCollection<?>> signals) Waits on the given signal collections.
static <T> Wait.OnSignal<T>	on(PCollection<?>... signals) Waits on the given signal collections.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Wait

public Wait()

Method Detail

on

public static <T> Wait.OnSignal<T> on(PCollection<?>... signals)

Waits on the given signal collections.

on

public static <T> Wait.OnSignal<T> on(java.util.List<PCollection<?>> signals)

Waits on the given signal collections.