org.apache.beam.sdk.transforms

Class Watch

java.lang.Object

org.apache.beam.sdk.transforms.Watch

@Experimental(value=SPLITTABLE_DO_FN)
public class Watch
extends java.lang.Object

Given a "poll function" that produces a potentially growing set of outputs for an input, this transform simultaneously continuously watches the growth of output sets of all inputs, until a per-input termination condition is reached.

The output is returned as an unbounded PCollection of KV<InputT, OutputT>, where each OutputT is associated with the InputT that produced it, and is assigned with the timestamp that the poll function returned when this output was detected for the first time.

Hypothetical usage example for watching new files in a collection of directories, where for each directory we assume that new files will not appear if the directory contains a file named ".complete":

 PCollection<String> directories = ...;  // E.g. Create.of(single directory)
 PCollection<KV<String, String>> matches = filepatterns.apply(Watch.<String, String>growthOf(
   new PollFn<String, String>() {
     public PollResult<String> apply(TimestampedValue<String> input) {
       String directory = input.getValue();
       List<TimestampedValue<String>> outputs = new ArrayList<>();
       ... List the directory and get creation times of all files ...
       boolean isComplete = ... does a file ".complete" exist in the directory ...
       return isComplete ? PollResult.complete(outputs) : PollResult.incomplete(outputs);
     }
   })
   // Poll each directory every 5 seconds
   .withPollInterval(Duration.standardSeconds(5))
   // Stop watching each directory 12 hours after it's seen even if it's incomplete
   .withTerminationPerInput(afterTotalOf(Duration.standardHours(12)));
 

By default, the watermark for a particular input is computed from a poll result as "earliest timestamp of new elements in this poll result". It can also be set explicitly via Watch.Growth.PollResult.withWatermark(org.joda.time.Instant) if the Watch.Growth.PollFn can provide a more optimistic estimate.

Note: This transform works only in runners supporting Splittable DoFn: see capability matrix.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Watch.Growth<InputT,OutputT> Implementation of growthOf(org.apache.beam.sdk.transforms.Watch.Growth.PollFn<InputT, OutputT>).

Constructor Summary

Constructors

Constructor and Description
Watch()

Method Summary

Modifier and Type	Method and Description
static <InputT,OutputT> Watch.Growth<InputT,OutputT>	growthOf(Watch.Growth.PollFn<InputT,OutputT> pollFn) Watches the growth of the given poll function.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Watch

public Watch()

Method Detail

growthOf

public static <InputT,OutputT> Watch.Growth<InputT,OutputT> growthOf(Watch.Growth.PollFn<InputT,OutputT> pollFn)

Watches the growth of the given poll function. See class documentation for more details.