apache_beam.testing.util module
Utilities for testing Beam pipelines.
- apache_beam.testing.util.assert_that(actual, matcher, label='assert_that', reify_windows=False, use_global_window=True)[source]
A PTransform that checks a PCollection has an expected value.
Note that assert_that should be used only for testing pipelines since the check relies on materializing the entire PCollection being checked.
- Parameters:
actual – A PCollection.
matcher – A matcher function taking as argument the actual value of a materialized PCollection. The matcher validates this actual value against expectations and raises BeamAssertException if they are not met.
label – Optional string label. This is needed in case several assert_that transforms are introduced in the same pipeline.
reify_windows – If True, matcher is passed a list of TestWindowedValue.
use_global_window – If False, matcher is passed a dictionary of (k, v) = (window, elements in the window).
- Returns:
Ignored.
- apache_beam.testing.util.equal_to_approx(expected, rel_tol=1e-09, abs_tol=0.0)[source]
Matcher used by assert_that that compares numeric elements approximately.
Behaves similarly to equal_to for sequence ordering and membership, but any real number elements (integers and floats) are compared using math.isclose instead of exact equality.
Approximate comparisons are also applied to real numbers nested within lists and tuples. Note that equal_to’s advanced handling for Beam Row and NamedTuple is not supported here. All other elements are compared using standard == equality.
- Parameters:
expected – The expected output or sequence to compare against.
rel_tol – The relative tolerance used by math.isclose (default: 1e-09).
abs_tol – The absolute tolerance used by math.isclose (default: 0.0).
Example
- assert_that(
pipeline | beam.Create([1.000000001, 2.0]), equal_to_approx([1.0, 2.0]))
- apache_beam.testing.util.equal_to_per_window(expected_window_to_elements)[source]
Matcher used by assert_that to check to assert expected windows.
The ‘assert_that’ statement must have reify_windows=True. This assertion works when elements are emitted and are finally checked at the end of the window.
- Parameters:
expected_window_to_elements – A dictionary where the keys are the windows to check and the values are the elements associated with each window.
- apache_beam.testing.util.is_not_empty()[source]
This is test method which makes sure that the pcol is not empty and it has some data in it. :return:
- apache_beam.testing.util.matches_all(expected)[source]
Matcher used by assert_that to check a set of matchers.
- Parameters:
expected – A list of elements or hamcrest matchers to be used to match the elements of a single PCollection.
- apache_beam.testing.util.open_shards(glob_pattern, mode='rt', encoding='utf-8')[source]
Returns a composite file of all shards matching the given glob pattern.
- Parameters:
glob_pattern (str) – Pattern used to match files which should be opened.
mode (str) – Specify the mode in which the file should be opened. For available modes, check io.open() documentation.
encoding (str) – Name of the encoding used to decode or encode the file. This should only be used in text mode.
- Returns:
A stream with the contents of the opened files.
- apache_beam.testing.util.row_namedtuple_equals_fn(expected, actual, fallback_equals_fn=None)[source]
equals_fn which can be used by equal_to which treats Rows and NamedTuples as equivalent types. This can be useful since Beam converts Rows to NamedTuples when they are sent across portability layers, so a Row may be converted to a NamedTuple automatically by Beam.
- class apache_beam.testing.util.TestWindowedValue(value, timestamp, windows, pane_info)[source]
Bases:
NamedTupleCreate new instance of TestWindowedValue(value, timestamp, windows, pane_info)
- pane_info: PaneInfo
Alias for field number 3