org.apache.beam.sdk.transforms

Class Watch

java.lang.Object

org.apache.beam.sdk.transforms.Watch

public class Watch
extends java.lang.Object

Given a "poll function" that produces a potentially growing set of outputs for an input, this transform simultaneously continuously watches the growth of output sets of all inputs, until a per-input termination condition is reached.

The output is returned as an unbounded PCollection of KV<InputT, OutputT>, where each OutputT is associated with the InputT that produced it, and is assigned with the timestamp that the poll function returned when this output was detected for the first time.

Hypothetical usage example for watching new files in a collection of directories, where for each directory we assume that new files will not appear if the directory contains a file named ".complete":

 PCollection<String> directories = ...;  // E.g. Create.of(single directory)
 PCollection<KV<String, String>> matches = filepatterns.apply(Watch.<String, String>growthOf(
   new PollFn<String, String>() {
     public PollResult<String> apply(TimestampedValue<String> input) {
       String directory = input.getValue();
       List<TimestampedValue<String>> outputs = new ArrayList<>();
       ... List the directory and get creation times of all files ...
       boolean isComplete = ... does a file ".complete" exist in the directory ...
       return isComplete ? PollResult.complete(outputs) : PollResult.incomplete(outputs);
     }
   })
   // Poll each directory every 5 seconds
   .withPollInterval(Duration.standardSeconds(5))
   // Stop watching each directory 12 hours after it's seen even if it's incomplete
   .withTerminationPerInput(afterTotalOf(Duration.standardHours(12)));
 

By default, the watermark for a particular input is computed from a poll result as "earliest timestamp of new elements in this poll result". It can also be set explicitly via Watch.Growth.PollResult.withWatermark(org.joda.time.Instant) if the Watch.Growth.PollFn can provide a more optimistic estimate.

Note: This transform works only in runners supporting Splittable DoFn: see capability matrix.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Watch.Growth<InputT,OutputT,KeyT> Implementation of growthOf(org.apache.beam.sdk.transforms.Watch.Growth.PollFn<InputT, OutputT>, org.apache.beam.sdk.transforms.Requirements).
protected static class	Watch.WatchGrowthFn<InputT,OutputT,KeyT,TerminationStateT>

Constructor Summary

Constructors

Constructor and Description
Watch()

Method Summary

Modifier and Type	Method and Description
static <InputT,OutputT,KeyT> Watch.Growth<InputT,OutputT,KeyT>	growthOf(Contextful<Watch.Growth.PollFn<InputT,OutputT>> pollFn, SerializableFunction<OutputT,KeyT> outputKeyFn) Watches the growth of the given poll function, using the given "key function" to deduplicate outputs.
static <InputT,OutputT> Watch.Growth<InputT,OutputT,OutputT>	growthOf(Watch.Growth.PollFn<InputT,OutputT> pollFn) Watches the growth of the given poll function.
static <InputT,OutputT> Watch.Growth<InputT,OutputT,OutputT>	growthOf(Watch.Growth.PollFn<InputT,OutputT> pollFn, Requirements requirements) Watches the growth of the given poll function.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Watch

public Watch()

Method Detail

growthOf

public static <InputT,OutputT> Watch.Growth<InputT,OutputT,OutputT> growthOf(Watch.Growth.PollFn<InputT,OutputT> pollFn,
                                                                             Requirements requirements)

Watches the growth of the given poll function. See class documentation for more details.

growthOf

public static <InputT,OutputT> Watch.Growth<InputT,OutputT,OutputT> growthOf(Watch.Growth.PollFn<InputT,OutputT> pollFn)

Watches the growth of the given poll function. See class documentation for more details.

growthOf

public static <InputT,OutputT,KeyT> Watch.Growth<InputT,OutputT,KeyT> growthOf(Contextful<Watch.Growth.PollFn<InputT,OutputT>> pollFn,
                                                                               SerializableFunction<OutputT,KeyT> outputKeyFn)

Watches the growth of the given poll function, using the given "key function" to deduplicate outputs. For example, if OutputT is a filename + file size, this can be a function that returns just the filename, so that if the same file is observed multiple times with different sizes, only the first observation is emitted.

By default, this is the identity function, i.e. the output is used as its own key.