apache_beam.testing.util module¶
Utilities for testing Beam pipelines.
-
class
apache_beam.testing.util.
TestWindowedValue
(value, timestamp, windows)¶ Bases:
tuple
Create new instance of TestWindowedValue(value, timestamp, windows)
-
timestamp
¶ Alias for field number 1
-
value
¶ Alias for field number 0
-
windows
¶ Alias for field number 2
-
-
apache_beam.testing.util.
equal_to_per_window
(expected_window_to_elements)[source]¶ Matcher used by assert_that to check to assert expected windows.
The ‘assert_that’ statement must have reify_windows=True. This assertion works when elements are emitted and are finally checked at the end of the window.
Parameters: expected_window_to_elements – A dictionary where the keys are the windows to check and the values are the elements associated with each window.
-
apache_beam.testing.util.
matches_all
(expected)[source]¶ Matcher used by assert_that to check a set of matchers.
Parameters: expected – A list of elements or hamcrest matchers to be used to match the elements of a single PCollection.
-
apache_beam.testing.util.
is_not_empty
()[source]¶ This is test method which makes sure that the pcol is not empty and it has some data in it. :return:
-
apache_beam.testing.util.
assert_that
(actual, matcher, label='assert_that', reify_windows=False, use_global_window=True)[source]¶ A PTransform that checks a PCollection has an expected value.
Note that assert_that should be used only for testing pipelines since the check relies on materializing the entire PCollection being checked.
Parameters: - actual – A PCollection.
- matcher – A matcher function taking as argument the actual value of a materialized PCollection. The matcher validates this actual value against expectations and raises BeamAssertException if they are not met.
- label – Optional string label. This is needed in case several assert_that transforms are introduced in the same pipeline.
- reify_windows – If True, matcher is passed a list of TestWindowedValue.
- use_global_window – If False, matcher is passed a dictionary of (k, v) = (window, elements in the window).
Returns: Ignored.
-
apache_beam.testing.util.
open_shards
(glob_pattern, mode='rt', encoding='utf-8')[source]¶ Returns a composite file of all shards matching the given glob pattern.
Parameters: - glob_pattern (str) – Pattern used to match files which should be opened.
- mode (str) – Specify the mode in which the file should be opened. For available modes, check io.open() documentation.
- encoding (str) – Name of the encoding used to decode or encode the file. This should only be used in text mode.
Returns: A stream with the contents of the opened files.