We’re happy to announce that we’re providing new Beam starter projects! 🎉

Setting up and configuring a new project can be time consuming, and varies in different languages. We hope this will make it easier for you to get started in creating new Apache Beam projects and pipelines.

All the starter projects come in their own GitHub repository, so you can simply clone a repo and you’re ready to go. Each project comes with a README with how to use it, a simple “Hello World” pipeline, and a test for the pipeline. The GitHub repositories come pre-configured with GitHub Actions to automatically run tests when pull requests are opened or modified, and Dependabot is enabled to make sure all the dependencies are up to date. This all comes out of the box, so you can start playing with your Beam pipeline without a hassle.

For example, here’s how to get started with Java:

git clone https://github.com/apache/beam-starter-java
cd beam-starter-java

# Install Java and Gradle with sdkman.
curl -s "https://get.sdkman.io" | bash
sdk install java 11.0.12-tem
sdk install gradle

# To run the pipeline.
gradle run

# To run the tests.
gradle test

And here’s how to get started with Python:

git clone https://github.com/apache/beam-starter-python
cd beam-starter-python

# Set up a virtual environment with the dependencies.
python -m venv env
source env/bin/activate
pip install -r requirements.txt

# To run the pipeline.
python main.py

# To run the tests.
python -m unittest

Here are the starter projects; you can choose your favorite language:

• [Java] github.com/apache/beam-starter-java – Includes both Gradle and Maven configurations.
• [Python] github.com/apache/beam-starter-python – Includes a setup.py file to allow multiple files in your pipeline.
• [Go] github.com/apache/beam-starter-go – Includes how to register different types of functions for ParDo.
• [Kotlin] github.com/apache/beam-starter-kotlin – Adapted to idiomatic Kotlin
• [Scala] github.com/apache/beam-starter-scala – Coming soon!

We have updated the Java quickstart to use the new starter project, and we’re working on updating the Python and Go quickstarts as well.

We hope you find this useful. Feedback and contributions are always welcome! So feel free to create a GitHub issue, or open a Pull Request to any of the starter project repositories.