apache_beam.testing.pipeline_verifiers module

End-to-end test result verifiers

A set of verifiers that are used in end-to-end tests to verify state/output of test pipeline job. Customized verifier should extend hamcrest.core.base_matcher.BaseMatcher and override _matches.

class apache_beam.testing.pipeline_verifiers.PipelineStateMatcher(expected_state='DONE')

Bases: BaseMatcher

Matcher that verify pipeline job terminated in expected state

Matcher compares the actual pipeline terminate state with expected. By default, PipelineState.DONE is used as expected state.

describe_to(description)

describe_mismatch(pipeline_result, mismatch_description)

class apache_beam.testing.pipeline_verifiers.FileChecksumMatcher(file_path, expected_checksum, sleep_secs=None)

Bases: BaseMatcher

Matcher that verifies file(s) content by comparing file checksum.

Use apache_beam.io.filebasedsink to fetch file(s) from given path. File checksum is a hash string computed from content of file(s).

Initialize a FileChecksumMatcher object

Parameters:

• file_path – A string that is the full path of output file. This path can contain globs.

• expected_checksum – A hash string that is computed from expected result.

• sleep_secs – Number of seconds to wait before verification start. Extra time are given to make sure output files are ready on FS.

describe_to(description)

describe_mismatch(pipeline_result, mismatch_description)

apache_beam.testing.pipeline_verifiers.retry_on_io_error_and_server_error(exception)

Filter allowing retries on file I/O errors and service error.