apache_beam.io.watermark_estimators module

A collection of WatermarkEstimator implementations that SplittableDoFns can use.

class apache_beam.io.watermark_estimators.MonotonicWatermarkEstimator(timestamp)

Bases: apache_beam.io.iobase.WatermarkEstimator

A WatermarkEstimator which assumes that timestamps of all ouput records are increasing monotonically.

For a new <element, restriction> pair, the initial value is None. When resuming processing, the initial timestamp will be the last reported watermark.

observe_timestamp(timestamp)

current_watermark()

get_estimator_state()

static default_provider()

Provide a default WatermarkEstimatorProvider for MonotonicWatermarkEstimator.

class apache_beam.io.watermark_estimators.WalltimeWatermarkEstimator(timestamp=None)

Bases: apache_beam.io.iobase.WatermarkEstimator

A WatermarkEstimator which uses processing time as the estimated watermark.

observe_timestamp(timestamp)

current_watermark()

get_estimator_state()

static default_provider()

Provide a default WatermarkEstimatorProvider for WalltimeWatermarkEstimator.

class apache_beam.io.watermark_estimators.ManualWatermarkEstimator(watermark)

Bases: apache_beam.io.iobase.WatermarkEstimator

A WatermarkEstimator which is controlled manually from within a DoFn.

The DoFn must invoke set_watermark to advance the watermark.

observe_timestamp(timestamp)

current_watermark()

get_estimator_state()

set_watermark(timestamp)

Sets a timestamp before or at the timestamps of all future elements produced by the associated DoFn.

This can be approximate. If records are output that violate this guarantee, they will be considered late, which will affect how they will be processed. See https://beam.apache.org/documentation/programming-guide/#watermarks-and-late-data for more information on late data and how to handle it.

However, this value should be as late as possible. Downstream windows may not be able to close until this watermark passes their end.

static default_provider()

Provide a default WatermarkEstimatorProvider for WalltimeWatermarkEstimator.