Package org.apache.beam.sdk.transforms

Class WithTimestamps<T>

java.lang.Object
org.apache.beam.sdk.transforms.PTransform<PCollection<T>,PCollection<T>>
org.apache.beam.sdk.transforms.WithTimestamps<T>
All Implemented Interfaces:
Serializable, HasDisplayData
public class WithTimestamps<T> extends PTransform<PCollection<T>,PCollection<T>>
A PTransform for assigning timestamps to all the elements of a PCollection.

Timestamps are used to assign Windows to elements within the Window.into(org.apache.beam.sdk.transforms.windowing.WindowFn) PTransform. Assigning timestamps is useful when the input data set comes from a Source without implicit timestamps (such as TextIO).

See Also:

  • Method Details

    • of

      public static <T> WithTimestamps<T> of(SerializableFunction<T,Instant> fn)
      For a SerializableFunction fn from T to Instant, outputs a PTransform that takes an input PCollection<T> and outputs a PCollection<T> containing every element v in the input where each element is output with a timestamp obtained as the result of fn.apply(v).

      If the input PCollection elements have timestamps, the output timestamp for each element must not be before the input element's timestamp minus the value of getAllowedTimestampSkew(). If an output timestamp is before this time, the transform will throw an IllegalArgumentException when executed. Use withAllowedTimestampSkew(Duration) to update the allowed skew.

      CAUTION: Use of withAllowedTimestampSkew(Duration) permits elements to be emitted behind the watermark. These elements are considered late, and if behind the allowed lateness of a downstream PCollection may be silently dropped. See https://github.com/apache/beam/issues/18065 for details on a replacement.

      Each output element will be in the same windows as the input element. If a new window based on the new output timestamp is desired, apply a new instance of Window.into(WindowFn).

      This transform will fail at execution time with a NullPointerException if for any input element the result of fn.apply(v) is null.

      Example of use in Java 8: 

      
 PCollection<Record> timestampedRecords = records.apply(
     WithTimestamps.of((Record rec) -> rec.getInstant());

    • withAllowedTimestampSkew

      @Deprecated public WithTimestamps<T> withAllowedTimestampSkew(Duration allowedTimestampSkew)
      Deprecated.
      This method permits a to elements to be emitted behind the watermark. These elements are considered late, and if behind the allowed lateness of a downstream PCollection may be silently dropped. See https://github.com/apache/beam/issues/18065 for details on a replacement.
      Return a new WithTimestamps like this one with updated allowed timestamp skew, which is the maximum duration that timestamps can be shifted backward. Does not modify this object.

      The default value is Duration.ZERO, allowing timestamps to only be shifted into the future. For infinite skew, use new Duration(Long.MAX_VALUE).

    • getAllowedTimestampSkew

      @Deprecated public Duration getAllowedTimestampSkew()
      Deprecated.
      This method permits a to elements to be emitted behind the watermark. These elements are considered late, and if behind the allowed lateness of a downstream PCollection may be silently dropped. See https://github.com/apache/beam/issues/18065 for details on a replacement.
      Returns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backwards from the timestamp of the input element.
      See Also:

    • expand

      public PCollection<T> expand(PCollection<T> input)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PCollection<T>,PCollection<T>>