We are happy to present the new 2.41.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.41.0, check out the detailed release notes.

I/Os

• Projection Pushdown optimizer is now on by default for streaming, matching the behavior of batch pipelines since 2.38.0. If you encounter a bug with the optimizer, please file an issue and disable the optimizer using pipeline option --experiments=disable_projection_pushdown.

New Features / Improvements

• Previously available in Java sdk, Python sdk now also supports logging level overrides per module. (#18222).

Breaking Changes

• Projection Pushdown optimizer may break Dataflow upgrade compatibility for optimized pipelines when it removes unused fields. If you need to upgrade and encounter a compatibility issue, disable the optimizer using pipeline option --experiments=disable_projection_pushdown.

Deprecations

• Support for Spark 2.4.x is deprecated and will be dropped with the release of Beam 2.44.0 or soon after (Spark runner) (#22094).
• The modules amazon-web-services and kinesis for AWS Java SDK v1 are deprecated in favor of amazon-web-services2 and will be eventually removed after a few Beam releases (Java) (#21249).

Bugfixes

• Fixed a condition where retrying queries would yield an incorrect cursor in the Java SDK Firestore Connector (#22089).
• Fixed plumbing allowed lateness in Go SDK. It was ignoring the user set value earlier and always used to set to 0. (#22474).

Known Issues

• See a full list of open issues that affect this version.

List of Contributors

According to git shortlog, the following people contributed to the 2.41.0 release. Thank you to all contributors!

Ahmed Abualsaud Ahmet Altay akashorabek Alexey Inkin Alexey Romanenko Anand Inguva andoni-guzman Andrew Pilloud Andrey Andy Ye Balázs Németh Benjamin Gonzalez BjornPrime Brian Hulette bulat safiullin bullet03 Byron Ellis Chamikara Jayalath Damon Douglas Daniel Oliveira Daniel Thevessen Danny McCormick David Huntsperger Dheeraj Gharde Etienne Chauchot Evan Galpin Fernando Morales Heejong Lee Jack McCluskey johnjcasey Kenneth Knowles Ke Wu Kiley Sok Liam Miller-Cushon Lucas Nogueira Luke Cwik MakarkinSAkvelon Manu Zhang Minbo Bae Moritz Mack Naireen Hussain Ning Kang Oleh Borysevych Pablo Estrada pablo rodriguez defino Pranav Bhandari Rebecca Szper Red Daly Reuven Lax Ritesh Ghorse Robert Bradshaw Robert Burke Ryan Thompson Sam Whittle Steven Niemitz Valentyn Tymofieiev Vincent Marquez Vitaly Terentyev Vlad Vladislav Chunikhin Yichi Zhang Yi Hu yirutang Yixiao Shen Yu Feng