Some larger features are developed on a feature branch before being merged into
master. In particular, this is often used for initial development of new
components like SDKs or runners.
We expect the work on a feature branch to be incomplete, but it must not
be lower quality. Code reviews for feature branches must have the same
standards as code reviews for
master. Once a feature branch is ready for
master, the set of changes will be too large to review in its
entirety. Because of this, the code reviews during development must be
thorough and trustworthy.
Establishing a feature branch
If your project is large enough to need a feature branch, there should be a discussion on the mailing list. The first step is to engage there to raise awareness that you want to start a large project. Almost any project should be accepted – there is no real cost to letting a feature branch exist – but you may find other interested contributors or gain other advice from the community.
After the community discussion, a committer must create your feature branch. Any committer can do create the branch through the GitHub UIs or by pushing directly to GitHub or ASF’s gitbox.
Developing on a feature branch
To contribute code on a feature branch, use the same process as in the
Contribution Guide, but
master with the name of the feature branch.
Since feature branches are often used for new components, you may find that there is no committer familiar with all the details of the new language or runner. In that case, consider asking someone else familiar with the technology to do an initial review before looping in a committer for a final review and merge.
If you are working on a feature branch, you’ll also want to frequently merge in
master. This prevents the feature branch from
deviating too far from the current state of
master. Like all changes, this
should be done via pull request. A committer may self-merge such a pull request
if there are no conflicts or test failures. If there are any conflicts or tests
that need fixing, then those should get a full review from another committer.
To merge a feature branch into
master, new components and major features
should meet the following guidelines.
- Have at least 2 contributors interested in maintaining it, and 1 committer interested in supporting it
- Provide both end-user and developer-facing documentation
- Have at least a basic level of unit test coverage
- Run all existing applicable integration tests with other Beam components and create additional tests as appropriate
Merging a new runner into
A new runner should:
- Be able to handle a subset of the model that addresses a significant set of use cases, such as ‘traditional batch’ or ‘processing time streaming’.
- Update the capability matrix with the current status
- Add a webpage under
Merging a new SDK into
A new SDK should:
- Provide the ability to construct graphs with all the basic building blocks of the model (ParDo, GroupByKey, Window, Trigger, etc)
- Begin fleshing out the common composite transforms (Count, Join, etc) and I/O connectors (Text, Kafka, etc)
- Have at least one runner that can execute the complete model (may be a direct runner)
- Provide integration tests for executing against current and future runners
- Add a webpage under