Contribution guide

Overview

Get Started

Connect with the Beam community

1. Consider subscribing to the dev@beam.apache.org mailing list, especially if you plan to make more than one change or the change will be large. All decisions are consensus-based and happen on the public mailing list.
2. (Optionally) Join the #beam channel of the ASF Slack.

Accounts and Permissions

• Beam Wiki Space: Anyone has read access. If you wish to contribute changes, please create an account and request edit access on the dev@beam.apache.org mailing list (include your Wiki account user ID).

• Pull requests can only be merged by a Beam committer.

• Voting on a release: Everyone can vote. Only Beam PMC members should mark their votes as binding.

Communication

All communication is expected to align with the Code of Conduct.

Discussion about contributing code to Beam happens on the dev@beam.apache.org mailing list. Introduce yourself!

Questions can be asked on the #beam channel of the ASF Slack. Introduce yourself!

Before You Begin

Prerequisites

• A GitHub account.
• A Linux, macOS, or Microsoft Windows development environment.
• Java JDK 8 installed.
• Docker installed for some tasks including building worker containers and testing changes to this website locally.
• For SDK Development:
  • Go 1.16.0 or later installed for Go SDK development.
  • Python 3.x interpreters. You will need Python interpreters for all Python versions supported by Beam. Interpreters should be installed and available in shell via python3.x commands. For more information, see: Python installation tips in Developer Wiki.
• For large contributions, a signed Individual Contributor License. Agreement (ICLA) to the Apache Software Foundation (ASF).

Share Your Intent

1. Find or create an issue in the Beam repo. Tracking your work in an issue will avoid duplicated or conflicting work, and provide a place for notes. Later, your pull request will be linked to the issue as well.
2. Comment “.take-issue” on the issue. This will cause the issue to be assigned to you. When you’ve completed the issue, you can close it by commenting “.close-issue”. If you are a committer and would like to assign an issue to a non-committer, they must comment on the issue first; please tag the user asking them to do so or to comment “`.take-issue`”. The command will be ignored if it is surrounded by ``` markdown characters.
3. If your change is large or it is your first change, it is a good idea to discuss it on the dev@beam.apache.org mailing list.
4. For large changes create a design doc (template, examples) and email it to the dev@beam.apache.org mailing list.

Setup Your Environment

Before you begin, check out the Wiki pages. There are many useful tips about Git, Go, Gradle, Java, Python, etc.

Configuration Options

You have two options for configuring your development environment:

• Local:
  • Manually installing the prerequisites.
  • Using the automated script for Linux and macOS.
• Container-based: using a Docker image.

Local: Debian-based Distribution

Manual steps

To install these in a Debian-based distribution:

1. Execute:

   sudo apt-get install \
      openjdk-8-jdk \
      python-setuptools \
      python-pip \
      virtualenv \
      tox \
      docker-ce
   

2. On some systems, like Ubuntu 20.04, install these:

   pip3 install grpcio-tools mypy-protobuf
   

3. If you develop in GO:
4. Install Go.
5. Check BEAM repo is in: $GOPATH/src/github.com/apache/
6. At the end, it should look like this: $GOPATH/src/github.com/apache/beam
7. Once Go is installed, install goavro:

   $ export GOPATH=`pwd`/sdks/go/examples/.gogradle/project_gopath
   $ go get github.com/linkedin/goavro
   

Important: gLinux users should configure their machines for sudoless Docker.

Automated script for Linux and macOS

You can install these in a Debian-based distribution for Linux or macOs using the local-env-setup.sh script, which is part of the Beam repo. It contains:

• pip3 packages
• go packages
• goavro
• JDK 8
• Python
• Docker

To install execute:

./local-env-setup.sh

Container: Docker-based

Alternatively, you can use the Docker based local development environment to wrap your clone of the Beam repo into a container meeting the requirements above.

You can start this container using the start-build-env.sh script which is part of the Beam repo.

Execute:

./start-build-env.sh

Development Setup

1. Check Git workflow tips if you need help with git forking, cloning, branching, committing, pull requests, and squashing commits.

2. Make a fork of https://github.com/apache/beam repo.

3. Clone the forked repository. You can download it anywhere you like.

   $ mkdir -p ~/path/to/your/folder
   $ cd ~/path/to/your/folder
   $ git clone https://github.com/forked/apache/beam
   $ cd beam
   

   For Go development:

   We recommend putting it in your $GOPATH ($HOME/go by default on Unix systems).

   Clone the repo, and update your branch as normal:

   $ git clone https://github.com/apache/beam.git
   $ cd beam
   $ git remote add <GitHub_user> git@github.com:<GitHub_user>/beam.git
   $ git fetch --all
   

   Get or Update all the Go SDK dependencies:

   $ go get -u ./...
   

4. Check the environment was set up correctly.

   Option 1: validate the Go, Java, and Python environments:

   Important: Make sure you have activated Python development.

   ./gradlew :checkSetup
   

   Option 2: Run independent checks:

• For Go development:

  export GOLANG_PROTOBUF_REGISTRATION_CONFLICT=ignore./gradlew :sdks:go:examples:wordCount
  

• For Python development:

  ./gradlew :sdks:python:wordCount
  

• For Java development:

  ./gradlew :examples:java:wordCount
  

5. Familiarize yourself with gradle and the project structure.

   At the root of the git repository, run:

   $ ./gradlew projects
   

   Examine the available tasks in a project. For the default set of tasks, use:

   $ ./gradlew tasks
   

   For a given module, use:

   $ ./gradlew -p sdks/java/io/cassandra tasks
   

   For an exhaustive list of tasks, use:

   $ ./gradlew tasks --all
   

6. Make sure you can build and run tests.

   Since Beam is a large project, usually, you will want to limit testing to the particular module you are working on. Gradle will build just the necessary things to run those tests. For example:

   $ ./gradlew -p sdks/go check
   $ ./gradlew -p sdks/java/io/cassandra check
   $ ./gradlew -p runners/flink check
   

7. Now you may want to set up your preferred IDE and other aspects of your development environment. See the Developers’ wiki for tips, guides, and FAQs on:

• IntelliJ
• Java
• Python
• Go
• Website
• Gradle
• Jenkins
• FAQ

Contribute Code

Find Efforts to Contribute to

A great way to contribute is to join an existing effort. If you want to get involved but don’t have a project in mind, check our list of open starter tasks. For the most intensive efforts, check out the roadmap.

Additional Resources

Please see Beam developers’ Wiki Contributor FAQ for more information.

If you are contributing a PTransform to Beam, we have an extensive PTransform Style Guide.

If you are contributing a Runner to Beam, refer to the Runner authoring guide.

Review design documents.

You can also find out more information on the Beam developers’ Wiki.