blog & release
Apache Beam 2.44.0Kenneth Knowles [@KennKnowles]
We are happy to present the new 2.44.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.
For more information on changes in 2.44.0, check out the detailed release notes.
- Support for Bigtable sink (Write and WriteBatch) added (Go) (#23324).
- S3 implementation of the Beam filesystem (Go) (#23991).
- Support for SingleStoreDB source and sink added (Java) (#22617).
- Added support for DefaultAzureCredential authentication in Azure Filesystem (Python) (#24210).
- Added new CdapIO for CDAP Batch and Streaming Source/Sinks (Java) (#24961).
- Added new SparkReceiverIO for Spark Receivers 2.4.* (Java) (#24960).
New Features / Improvements
- Beam now provides a portable “runner” that can render pipeline graphs with
python -m apache_beam.runners.render --helpfor more details.
- Local packages can now be used as dependencies in the requirements.txt file, rather
than requiring them to be passed separately via the
--extra_packageoption (Python) (#23684).
- Pipeline Resource Hints now supported via
--resource_hintsflag (Go) (#23990).
- Make Python SDK containers reusable on portable runners by installing dependencies to temporary venvs (BEAM-12792).
- RunInference model handlers now support the specification of a custom inference function in Python (#22572)
- Support for
map_windowsurn added to Go SDK (#24307).
ParquetIO.withSplitwas removed since splittable reading has been the default behavior since 2.35.0. The effect of this change is to drop support for non-splittable reading (Java)(#23832).
beam-sdks-java-extensions-google-cloud-platform-coreis no longer a dependency of the Java SDK Harness. Some users of a portable runner (such as Dataflow Runner v2) may have an undeclared dependency on this package (for example using GCS with TextIO) and will now need to declare the dependency.
beam-sdks-java-coreis no longer a dependency of the Java SDK Harness. Users of a portable runner (such as Dataflow Runner v2) will need to provide this package and its dependencies.
- Slices now use the Beam Iterable Coder. This enables cross language use, but breaks pipeline updates if a Slice type is used as a PCollection element or State API element. (Go)#24339
- Fixed JmsIO acknowledgment issue (Java) (#20814)
- Fixed Beam SQL CalciteUtils (Java) and Cross-language JdbcIO (Python) did not support JDBC CHAR/VARCHAR, BINARY/VARBINARY logical types (#23747, #23526).
- Ensure iterated and emitted types are used with the generic register package are registered with the type and schema registries.(Go) (#23889)
List of Contributors
According to git shortlog, the following people contributed to the 2.44.0 release. Thank you to all contributors!
Elias Segundo Antonio
John J. Casey
Steven van Rossum