Pre-commit test policies
- Pre-commit test - Any single test in a pre-commit test suite.
- Pre-commit test suite - A collection of pre-commit tests that have a common denominator. A test suite runs in a single Jenkins job. Currently, suites are grouped by SDK languages, e.g., Python, Java, and Go.
- A PR must pass pre-commit tests before being committed to the main Beam repo.
- The relevant pre-commit test suites are automatically launched according to PR contents.
Breakage is when one or more tests in a pre-commit test suite fails or is flaky (occasionally fails).
- Breakages should be fixed within 8 hours.
Slowness is when the total time to run a pre-commit suite exceeds 30 minutes*, including the time the job spends in the Jenkins queue.
- Slowness should be fixed within 24 hours.
* See the Pre-commit Slowness Triage Guide for a precise definition of slowness and for information on dealing with slowness.
For any problem, the options are, one of:
- Roll back the culprit PR.
- Roll out a fix within 24 hours.
- Disable the slow test or feature temporarily (make sure there’s a tracking issue to re-enable it).