apache_beam.testing.test_stream module

Provides TestStream for verifying streaming runner semantics.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.testing.test_stream.Event

Bases: object

Test stream event to be emitted during execution of a TestStream.

to_runner_api(element_coder)

static from_runner_api(proto, element_coder)

class apache_beam.testing.test_stream.ElementEvent(timestamped_values, tag=None)

Bases: apache_beam.testing.test_stream.Event

Element-producing test stream event.

to_runner_api(element_coder)

class apache_beam.testing.test_stream.WatermarkEvent(new_watermark, tag=None)

Bases: apache_beam.testing.test_stream.Event

Watermark-advancing test stream event.

to_runner_api(unused_element_coder)

class apache_beam.testing.test_stream.ProcessingTimeEvent(advance_by)

Bases: apache_beam.testing.test_stream.Event

Processing time-advancing test stream event.

to_runner_api(unused_element_coder)

class apache_beam.testing.test_stream.TestStream(coder=FastPrimitivesCoder, events=None, output_tags=None, endpoint=None)

Bases: apache_beam.transforms.ptransform.PTransform

Test stream that generates events on an unbounded PCollection of elements.

Each event emits elements, advances the watermark or advances the processing time. After all of the specified elements are emitted, ceases to produce output.

Applying the PTransform will return a single PCollection if only the default output or only one output tag has been used. Otherwise a dictionary of output names to PCollections will be returned.

To use the multi-output functionality pelase use the ‘passthrough_pcollection_output_ids’ flag. See BEAM-9322 for more info.

To add flag:

from apache_beam.options.pipeline_options import DebugOptions
options = ...
options.view_as(DebugOptions).add_experiment(
    'passthrough_pcollection_output_ids')

Parameters:

• coder – (apache_beam.Coder) the coder to encode/decode elements.
• events – (List[Event]) a list of instructions for the TestStream to execute. If specified, the events tags must exist in the output_tags.
• output_tags – (List[str]) Initial set of outputs. If no event references an output tag, no output will be produced for that tag.
• endpoint – (str) a URL locating a TestStreamService.

get_windowing(unused_inputs)

expand(pbegin)

add_elements(elements, tag=None, event_timestamp=None)

Add elements to the TestStream.

Elements added to the TestStream will be produced during pipeline execution. These elements can be TimestampedValue, WindowedValue or raw unwrapped elements that are serializable using the TestStream’s specified Coder. When a TimestampedValue or a WindowedValue element is used, the timestamp of the TimestampedValue or WindowedValue will be the timestamp of the produced element; otherwise, the current watermark timestamp will be used for that element. The windows of a given WindowedValue are ignored by the TestStream.

advance_watermark_to(new_watermark, tag=None)

Advance the watermark to a given Unix timestamp.

The Unix timestamp value used must be later than the previous watermark value and should be given as an int, float or utils.timestamp.Timestamp object.

advance_watermark_to_infinity(tag=None)

Advance the watermark to the end of time, completing this TestStream.

advance_processing_time(advance_by)

Advance the current processing time by a given duration in seconds.

The duration must be a positive second duration and should be given as an int, float or utils.timestamp.Duration object.

to_runner_api_parameter(context)

static from_runner_api_parameter(ptransform, payload, context)