Contribution guide

Apache Beam is a very welcoming and collaborative community, and there are lots of opportunities to contribute, both code and non-code. You can, for example:

  • ask or answer questions on user@beam.apache.org or stackoverflow
  • review proposed design ideas on dev@beam.apache.org
  • improve the documentation
  • file bug reports
  • test releases
  • review changes
  • write new examples
  • improve your favorite language SDK (Java, Python, Go, etc)
  • improve specific runners (Apache Flink, Apache Spark, Google Cloud Dataflow, etc)
  • improve or add IO connectors
  • add new transform libraries (statistics, ML, image processing, etc)
  • work on the core programming model (read more about what a Beam pipeline is and how it runs here in Documentation)
  • improve the developer experience (for example, Windows guides)
  • add answers to the contribution FAQ
  • organize local meetups of users or contributors to Apache Beam

Below is a tutorial for contributing code to Beam, covering our tools and typical process in detail.

Get Started

Connect with the Beam community

  1. Consider subscribing to the dev@beam.apache.org mailing list, especially if you plan to make more than one change or the change will be large. All decisions are consensus-based and happen on the public mailing list.
  2. (Optionally) Join the #beam channel of the ASF Slack.
  3. Create an account on Beam issue tracker (JIRA) (anyone can do this).

Accounts and Permissions

Communication

All communication is expected to align with the Code of Conduct.

Discussion about contributing code to Beam happens on the dev@beam.apache.org mailing list. Introduce yourself!

Questions can be asked on the #beam channel of the ASF Slack. Introduce yourself!

Before You Begin

Prerequisites

Share Your Intent

  1. Find or create an issue in the Beam issue tracker (JIRA). Tracking your work in an issue will avoid duplicated or conflicting work, and provide a place for notes. Later, your pull request will be linked to the issue as well. You can find the Jira beginner’s guide and some useful Jira tips in our Wiki.
  2. Assign the issue to yourself. To get the permission to do so, email the dev@beam.apache.org mailing list to introduce yourself and to be added as a contributor in the Beam issue tracker including your ASF Jira Username. For example, see this welcome email.
  3. If your change is large or it is your first change, it is a good idea to discuss it on the dev@beam.apache.org mailing list.
  4. For large changes create a design doc (template, examples) and email it to the dev@beam.apache.org mailing list.

Setup Your Environment

Before you begin, check out the Wiki pages. There are many useful tips about Git, Go, Gradle, Java, Python, etc.

Configuration Options

You have two options for configuring your development environment:

Local: Debian-based Distribution
Manual steps

To install these in a Debian-based distribution:

  1. Execute:
    sudo apt-get install \
       openjdk-8-jdk \
       python-setuptools \
       python-pip \
       virtualenv \
       tox \
       docker-ce
    
  2. On some systems, like Ubuntu 20.04, install these:
    pip3 install grpcio-tools mypy-protobuf
    
  3. If you develop in GO:
  4. Install Go.
  5. Check BEAM repo is in: $GOPATH/src/github.com/apache/
  6. At the end, it should look like this: $GOPATH/src/github.com/apache/beam
  7. Once Go is installed, install goavro:
    $ export GOPATH=`pwd`/sdks/go/examples/.gogradle/project_gopath
    $ go get github.com/linkedin/goavro
    

Important: gLinux users should configure their machines for sudoless Docker.

Automated script for Linux and macOS

You can install these in a Debian-based distribution for Linux or macOs using the local-env-setup.sh script, which is part of the Beam repo. It contains:

To install execute:

./local-env-setup.sh
Container: Docker-based

Alternatively, you can use the Docker based local development environment to wrap your clone of the Beam repo into a container meeting the requirements above.

You can start this container using the start-build-env.sh script which is part of the Beam repo.

Execute:

./start-build-env.sh

Development Setup

  1. Check Git workflow tips if you need help with git forking, cloning, branching, committing, pull requests, and squashing commits.

  2. Make a fork of https://github.com/apache/beam repo.

  3. Clone the forked repository. You can download it anywhere you like.

    $ mkdir -p ~/path/to/your/folder
    $ cd ~/path/to/your/folder
    $ git clone https://github.com/forked/apache/beam
    $ cd beam
    

    For Go development:

    We recommend putting it in your $GOPATH ($HOME/go by default on Unix systems).

    Clone the repo, and update your branch as normal:

    $ git clone https://github.com/apache/beam.git
    $ cd beam
    $ git remote add <GitHub_user> git@github.com:<GitHub_user>/beam.git
    $ git fetch --all
    

    Get or Update all the Go SDK dependencies:

    $ go get -u ./...
    
  4. Check the environment was set up correctly.

    Option 1: validate the Go, Java, and Python environments:

    Important: Make sure you have activated Python development.

    ./gradlew :checkSetup
    

    Option 2: Run independent checks:

  1. Familiarize yourself with gradle and the project structure.

    At the root of the git repository, run:

    $ ./gradlew projects
    

    Examine the available tasks in a project. For the default set of tasks, use:

    $ ./gradlew tasks
    

    For a given module, use:

    $ ./gradlew -p sdks/java/io/cassandra tasks
    

    For an exhaustive list of tasks, use:

    $ ./gradlew tasks --all
    
  2. Make sure you can build and run tests.

    Since Beam is a large project, usually, you will want to limit testing to the particular module you are working on. Gradle will build just the necessary things to run those tests. For example:

    $ ./gradlew -p sdks/go check
    $ ./gradlew -p sdks/java/io/cassandra check
    $ ./gradlew -p runners/flink check
    
  3. Now you may want to set up your preferred IDE and other aspects of your development environment. See the Developers’ wiki for tips, guides, and FAQs on:

Contribute Code

Create a Pull Request

  1. Make your code change. Every source file needs to include the Apache license header. Every new dependency needs to have an open source license compatible with Apache.

  2. Add unit tests for your change.

  3. Use descriptive commit messages that make it easy to identify changes and provide a clear history.

  4. When your change is ready to be reviewed and merged, create a pull request.

  5. Format commit messages and the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue. This will automatically link the pull request to the issue.

  6. The pull request and any changes pushed to it will trigger pre-commit jobs. If a test fails and appears unrelated to your change, you can cause tests to be re-run by adding a single line comment on your PR:

    retest this please
    

Pull request template has a link to a catalog of trigger phrases that start various post-commit tests suites. Use these sparingly because post-commit tests consume shared development resources.

Review Process and Releases

Get Reviewed

  1. Pull requests can only be merged by a Beam committer. To find a committer for your area, either:
  • look in the OWNERS file of the directory where you changed files, or
  • look for similar code merges, or
  • ask on dev@beam.apache.org

Use R: @username in the pull request to notify a reviewer.

  1. If you don’t get any response in 3 business days, email the dev@beam.apache.org mailing list to ask for someone to look at your pull request.

Make the Reviewer’s Job Easier

  1. Provide context for your changes in the associated JIRA issue and/or PR description.

  2. Avoid huge mega-changes.

  3. Review feedback typically leads to follow-up changes. It is easier to review follow-up changes when they are added as additional “fixup” commits to the existing PR/branch. This allows reviewer(s) to track the incremental progress and focus on new changes, and keeps comment threads attached to the code. Please refrain from squashing new commits into reviewed commits before review is completed. Because squashing reviewed and unreviewed commits often makes it harder to see the difference between the review iterations, reviewers may ask you to unsquash new changes.

  4. After review is complete and the PR is accepted, fixup commits should be squashed (see Git workflow tips). Beam committers can squash all commits in the PR during merge, however if a PR has a mixture of independent changes that should not be squashed, and fixup commits, then the PR author should help squashing fixup commits to maintain a clean commit history.

Apache Beam Releases

Apache Beam makes minor releases every 6 weeks. Apache Beam has a calendar for cutting the next release branch. Your change needs to be checked into master before the release branch is cut to make the next release.

Stale Pull Requests

The community will close stale pull requests in order to keep the project healthy. A pull request becomes stale after its author fails to respond to actionable comments for 60 days. Author of a closed pull request is welcome to reopen the same pull request again in the future. The associated JIRAs will be unassigned from the author but will stay open.

Troubleshooting

If you run into any issues, check out the contribution FAQ or ask on the dev@ mailing list or #beam channel of the ASF Slack.

If you didn’t find the information you were looking for in this guide, please reach out to the Beam community.

Find Efforts to Contribute to

A great way to contribute is to join an existing effort. If you want to get involved but don’t have a project in mind, check our list of open starter tasks. For the most intensive efforts, check out the roadmap.

Additional Resources

Please see Beam developers’ Wiki Contributor FAQ for more information.

If you are contributing a PTransform to Beam, we have an extensive PTransform Style Guide.

If you are contributing a Runner to Beam, refer to the Runner authoring guide.

Review design documents.

You can also find out more information on the Beam developers’ Wiki.