WordCount quickstart for Python
This guide shows you how to set up your Python development environment, get the Apache Beam SDK for Python, and run an example pipeline.
If you’re interested in contributing to the Apache Beam Python codebase, see the Contribution Guide.
The Python SDK supports Python 3.8, 3.9, 3.10, 3.11 and 3.12. Beam 2.48.0 was the last release with support for Python 3.7.
Set up your environment
For details, see Set up your development environment.
Get Apache Beam
Create and activate a virtual environment
A virtual environment is a directory tree containing its own Python distribution. To create a virtual environment, run:
A virtual environment needs to be activated for each shell that is to use it. Activating it sets some environment variables that point to the virtual environment’s directories.
To activate a virtual environment in Bash, run:
That is, execute the activate
script under the virtual environment directory you created.
For instructions using other shells, see the venv documentation.
Download and install
Install the latest Python SDK from PyPI:
Extra requirements
The above installation will not install all the extra dependencies for using features like the Google Cloud Dataflow runner. Information on what extra packages are required for different features are highlighted below. It is possible to install multiple extra requirements using something like pip install 'apache-beam[feature1,feature2]'
.
- Google Cloud Platform
- Installation Command:
pip install 'apache-beam[gcp]'
- Required for:
- Google Cloud Dataflow Runner
- GCS IO
- Datastore IO
- BigQuery IO
- Installation Command:
- Amazon Web Services
- Installation Command:
pip install 'apache-beam[aws]'
- Required for I/O connectors interfacing with AWS
- Installation Command:
- Microsoft Azure
- Installation Command:
pip install 'apache-beam[azure]'
- Required for I/O connectors interfacing with Microsoft Azure
- Installation Command:
- Beam YAML API
- Installation Command:
pip install 'apache-beam[yaml]'
- Required for using Beam YAML API
- Installation Command:
- Beam YAML Dataframe API
- Installation Command:
pip install 'apache-beam[dataframe]'
- Required for using Beam Dataframe API
- Installation Command:
- Tests
- Installation Command:
pip install 'apache-beam[test]'
- Required for developing Beam and running unit tests
- Installation Command:
- Docs
- Installation Command:
pip install 'apache-beam[docs]'
- Required for generating API documentation using Sphinx
- Installation Command:
Execute a pipeline
The Apache Beam examples directory has many examples. All examples can be run locally by passing the required arguments described in the example script.
For example, run wordcount.py
with the following command:
# As part of the initial setup, install Google Cloud Platform specific extra components. Make sure you
# complete the setup steps at /documentation/runners/dataflow/#setup
pip install apache-beam[gcp]
python -m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--output gs://<your-gcs-bucket>/counts \
--runner DataflowRunner \
--project your-gcp-project \
--region your-gcp-region \
--temp_location gs://<your-gcs-bucket>/tmp/
After the pipeline completes, you can view the output files at your specified
output path. For example, if you specify /dir1/counts
for the --output
parameter, the pipeline writes the files to /dir1/
and names the files
sequentially in the format counts-0000-of-0001
.
Next Steps
- Learn more about the Beam SDK for Python and look through the Python SDK API reference.
- Get An Interactive Overview of Beam
- Walk through these WordCount examples in the WordCount Example Walkthrough.
- Take a self-paced tour through our Learning Resources.
- Dive in to some of our favorite Videos and Podcasts.
- Join the Beam users@ mailing list.
Please don’t hesitate to reach out if you encounter any issues!
Last updated on 2024/12/06
Have you found everything you were looking for?
Was it all useful and clear? Is there anything that you would like to change? Let us know!