Unrecoverable errors in Beam Python

Unrecoverable errors are issues that occur at job start-up time and prevent jobs from ever running successfully. The problem usually stems from a misconfiguration. This page provides context about common errors and troubleshooting information.

Job submission or Python runtime version mismatch

If the Python version that you use to submit your job doesn’t match the Python version used to build the worker container, the job doesn’t run. The job fails immediately after job submission.

To resolve this issue, ensure that the Python version used to submit the job matches the Python container version.

Dependency resolution failures with pip

During worker start-up, the worker might fail and, depending on the runner, try to restart.

Before workers accept work, dependencies are checked and installed in the worker container. If a pipeline requires dependencies not already present in the runtime environment, they are installed at this time. When a problem occurs during this process, you might encounter dependency resolution failures.

Examples of problems include the following:

To resolve this issue, before submitting your job, ensure that the dependency versions provided in your requirements.txt file exist and that you can install them locally.

Dependency version mismatches

When your pipeline has dependency version mismatches, you might see ModuleNotFound errors or AttributeError messages.

To resolve these problems, ensure that the required dependencies and their versions are the same at runtime and in the submission environment. To help you identify these issues, in Apache Beam 2.52.0 and later versions, debug logs specify the dependencies at both stages. For more information, see Control the dependencies the pipeline uses.