apache_beam.testing.util module

Utilities for testing Beam pipelines.

class apache_beam.testing.util.TestWindowedValue(value, timestamp, windows)

Bases: tuple

Create new instance of TestWindowedValue(value, timestamp, windows)

timestamp

Alias for field number 1

value

Alias for field number 0

windows

Alias for field number 2

apache_beam.testing.util.equal_to(expected)[source]
apache_beam.testing.util.matches_all(expected)[source]

Matcher used by assert_that to check a set of matchers.

Parameters:expected – A list of elements or hamcrest matchers to be used to match the elements of a single PCollection.
apache_beam.testing.util.is_empty()[source]
apache_beam.testing.util.is_not_empty()[source]

This is test method which makes sure that the pcol is not empty and it has some data in it. :return:

apache_beam.testing.util.assert_that(actual, matcher, label='assert_that', reify_windows=False, use_global_window=True)[source]

A PTransform that checks a PCollection has an expected value.

Note that assert_that should be used only for testing pipelines since the check relies on materializing the entire PCollection being checked.

Parameters:
  • actual – A PCollection.
  • matcher – A matcher function taking as argument the actual value of a materialized PCollection. The matcher validates this actual value against expectations and raises BeamAssertException if they are not met.
  • label – Optional string label. This is needed in case several assert_that transforms are introduced in the same pipeline.
  • reify_windows – If True, matcher is passed a list of TestWindowedValue.
  • use_global_window – If False, matcher is passed a dictionary of (k, v) = (window, elements in the window).
Returns:

Ignored.
apache_beam.testing.util.open_shards(glob_pattern, mode='rt', encoding='utf-8')[source]

Returns a composite file of all shards matching the given glob pattern.

Parameters:
  • glob_pattern (str) – Pattern used to match files which should be opened.
  • mode (str) – Specify the mode in which the file should be opened. For available modes, check io.open() documentation.
  • encoding (str) – Name of the encoding used to decode or encode the file. This should only be used in text mode.
Returns:

A stream with the contents of the opened files.