Apache Beam

Apache Beam 2.77.0

Wed, 07 Oct 2026 14:00:00 -0500

We are happy to present the new 2.77.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.77.0, check out the detailed release notes.

Highlights

I/Os

Added schema_update_options to WriteToBigQuery file loads, allowing BigQuery load jobs to add nullable fields or relax required fields when appending data (Python) (#21141).
BigQueryIO now supports reading BigQuery Lakehouse runtime catalog (BigLake metastore) Iceberg tables with the Storage Read API, using 4-part project.catalog.namespace.table identifiers (or a TableReference with a composite catalog.namespace dataset id). Previously such references were silently mis-parsed (Java) (#39597) .
SolaceIO now supports reading and writing binary and text content data payload (Java) (#39875).
ClickHouseIO: support writing Decimal(P, S) / Decimal32/64/128/256 columns (Java) (#39840).
SolaceIO now supports reading and writing user properties (message metadata) (Java) (#40099).
[IcebergIO] AddFiles (IcebergAddFiles in YAML) can evolve the table schema before registering files, with schema_evolution_options, required_columns, incompatible_schema_handling and unverifiable_file_handling (Java/YAML, batch only) (#40144).

New Features / Improvements

(Java/Python) Watch can bound its deduplication state by event time, retiring an output key once the greatest emitted timestamp has moved more than the allowed lateness past it. Java adds Watch.growthOf(...).withTimestampCursor(). Python adds allowed_lateness for the existing timestamp_cursor option (#18459).
(Java) Spark Structured Streaming runner: stateful ParDo with state, timers, @RequiresTimeSortedInput and tagged outputs is now supported in batch mode (#39779).
(Python) Added support for Vertex AI Model Monitoring V2 in RunInference (#39738).
[Flink Runner] Added opt-in static round-robin split assignment for small bounded sources via the new sourceStaticSplitThresholdMb pipeline option. The default of 0 keeps the existing lazy pull-based assignment (#39873).
Added automatic caching of bounded, single-pane side-input views for classic Java Flink DataStream execution (#39866).
(Python) Added Sample.Any, the Python equivalent of Java’s Sample.any, which returns up to n arbitrary elements from a PCollection (#18552).

Breaking Changes

Portable Java SDK now encodes SchemaCoders in a portable way (#34672).
- Original custom Java coder encoding can still be obtained using StreamingOptions.setUpdateCompatibilityVersion(“2.76”) (#34672).
- Fixes (#36496), (#30276), (#29245).
(Python) TensorRTEngineHandlerNumPy now requires TensorRT 10 or later. TensorRT 8.x is no longer supported, since TensorRT 10 removed the engine binding API the handler was written against (#36306).
- Engines serialized by TensorRT 8.x must be rebuilt, as an engine can only be deserialized by the major version that built it.
- TensorRT 10 and later require a GPU with compute capability 7.5 or higher, which excludes NVIDIA Pascal and Volta GPUs.
- If dropping TensorRT 8.x support is a hard blocker for you, please comment on (#36306).

Bugfixes

(Java) Fixed the Spark runner firing processing-time timers in reverse timestamp order (#39824).
(Java) Fixed the Spark runner dropping the stored watermark of a streaming source with no update in a batch (#39822).
(Python) Fixed incorrect profiler options handling on portable runners (#39613).
(Java) KafkaIO dynamic reads no longer require the obsolete beam_fn_api experiment (#29998).
(Prism) Self-checkpointing splittable DoFns now resume after their requested delay instead of immediately, so polling SDFs no longer busy-spin (#39848).
(Java) MongoDbIO read splitting now preserves non-ObjectId _id types (e.g. string ids) instead of failing to parse the generated range filters (#39900).
(Go) Fixed GCS glob matching silently dropping objects when the glob pattern contains multi-byte characters (#39969).
(Python) Fixed TensorRTEngineHandlerNumPy failing with CUDA_ERROR_INVALID_VALUE on models with a single-element input or output tensor (#36306).
(Python) Fixed PickleCoder/_MemoizingPickleCoder.as_deterministic_coder() raising TypeError instead of returning a working deterministic coder (#28558).

According to git shortlog, the following people contributed to the 2.77.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Aditya Narayan, Ahmed Abualsaud, Alex Bevilacqua, Alexander Pochill, Ali Ebrahim, Andrew Crites, Arun Pandian, Ashwin S, Bruno Volpato, Chamikara Jayalath, Chris Gavin, Claire McGinty, Danny McCormick, Derrick Williams, Eiji Ogiwara, Elia Liu, Fabian Loris, Goutam Adwant, HansMarcus01, Israel Herraiz, Jack McCluskey, Jan Lukavský, Jeremy Schoemaker, Kenneth Knowles, Lalit Yadav, Lawrence Qiu, M Junaid Shaukat, Makoto Nagai, Maksym Tymoshyk, Manvith Panyam, Mattie Fu, Michael Gruschke, Mukesh Bhandarkar, Nicolas Gibanel, Paulius Kuzmickas, Radosław Stankiewicz, Ryan Wigglesworth, Sam Whittle, Sharan Teja M, Shizuma5, Shunping Huang, SreeramaYeshwanthGowd, Tobias Kaymak, Tom Newton, Udit Jain, Vitaly Terentyev, Yi Hu, ZIHAN DAI, akshayjadiyanv, claudevdm, darshan-sj, feefs, junaiddshaukat, kellen, nitinware, parveensania, tvalentyn

Beam modernization at project 10-year mark

Wed, 30 Sep 2026 20:00:01 -0800

Summary

This blog summarizes recent modernization efforts across the Apache Beam codebase mainly between Beam 2.68.0 and 2.75.0, such as major package upgrades, aligning APIs with new programming language standards, deprecations, etc. Note that new features are intentionally not included in the scope of this post, as it focuses on the modernization effort of existing components. Thank you to all contributors!

Originally part of the “Beam 3.0” proposal, systematic modernization efforts were planned towards Beam 2.75.0. As the community agreed on delivering improvements incrementally while maximally preserving compatibility, we instead named 2.75.0 the “10-year anniversary release”—marking 10 years since the first release (v0.1.0-incubating).

Package Upgrades

Core languages

[Java] Java 25 Support (2.69.0) and dropped Java 8 support (2.74.0)

Java 8 has been supported since Beam 2.3.0. Since 2025, the ecosystem has shifted to Java 11 and Java 17 for certain core dependencies. Because Java 8 remained the baseline language version for Beam for a long time, a phased transition was conducted in subsequent Beam versions:

Introduced a framework to allow different Beam components with different Java version compatibility (2.66.0, 2.67.0).
Stopped publishing Java 8 SDK container image. Java 8 pipeline execution uses Java 11 container (2.66.0).
IO expansion service (2.68.0) and certain components (IcebergIO, HCatalogIO) started moving to Java 11.

And finally, the Beam baseline moved to Java 11. As of Beam 2.75.0, certain Beam components require Java 17:

IcebergIO
DebeziumIO
DeltaIO
IO expansion service
Spark 4 runner

[Python] Python 3.13 and 3.14 Support (2.69.0, 2.73.0) and removed Python 3.9 support (2.70.0)

[Go] Minimum Go version updated to 1.26 (2.69.0, 2.73.0)

[SQL] Upgraded Beam vendored Calcite from 1.28.0 to 1.40.0 for Beam SQL (2.68.0) and removed ZetaSQL support (2.68.0)

Beam SQL was on Calcite 1.28.0 since Beam 2.35.0 (2021). As part of the Lakehouse initiative, Beam SQL has received renewed attention and moved to Calcite 1.40.0. A notable new feature is support for dialects such as BigQuery, SparkSQL, etc.

Here is an example of using a PostgreSQL dialect-provided scalar function in Beam YAML:

pipeline:
 transforms:
 - type: Create
 name: CreateSampleData
 config:
 elements:
 - {id: 1, tags: "java python go"}
 - {id: 2, tags: "rust cpp"}
 - {id: 3, tags: "javascript typescript"}
 - type: Sql
 name: TransformWithPostgresFunction
 input: CreateSampleData
 config:
 query: "SELECT id, STRING_TO_ARRAY(tags, ' ') as tag_list FROM PCOLLECTION"
 - type: LogForTesting
 input: TransformWithPostgresFunction

options:
 calcite_connection_properties: {"fun": "postgresql"}

ZetaSQL support was removed; Calcite SQL with the BigQuery dialect is the preferred alternative within Beam, or migrating to use BigQuery directly is also recommended.

Core dependencies

[Java] Upgraded Avro to 1.12 (2.74.0)

Avro 1.12 was released back in August 2024 and dropped Java 8 support. As part of Java 8 support removal, Beam upgraded its Avro dependency to 1.12 in Beam 2.74.0.

[Java] Upgraded GCS connector (gcsio) to 3.x (2.74.0)

GCS connector 3.0 was released back in December 2023 and dropped Java 8 support. As part of Java 8 support removal, Beam upgraded its GCS connector dependency to 3.1 in Beam 2.74.0. Note that there is a performance regression related to this upgrade in Beam 2.74.0 and 2.75.0 affecting pipelines with a moderate to heavy Cloud Storage read workload. To resolve this, upgrading to Beam 2.76.0 is recommended.

[Python] Protobuf 6.x Support (2.69.0)

[Python] Migrated default pickler to cloudpickle and made dill optional (2.68.0, 2.69.0)

Previously, the Python SDK relied on a pinned dill==0.3.1.1 dependency for serialization, which frequently caused dependency conflicts and blocked new Python version upgrades. After switching the default pickler to cloudpickle, Beam migrated the deterministic fallback coder for complex types (NamedTuple, Enum, dataclass) to cloudpickle in Beam 2.68.0 and moved dill out of the core requirements into an optional extra (apache-beam[dill]) in Beam 2.69.0.

[Python] Migrating GCP clients off `google-apitools` (2.72.0, 2.75.0)

The Python SDK has been migrating its internal GCP service integrations off the legacy google-apitools library and generated V1 clients to official google-cloud-* client libraries:

Migrated Cloud Build client in container builder to google-cloud-build (2.72.0) and removed standalone apitools HttpError usages (2.72.0).
Migrated Dataflow runner job submission and metrics client from generated dataflow_v1b3 apitools bindings to google-cloud-dataflow-client (2.75.0).

BigQuery is now the last remaining google-apitools client in the Python SDK, and its migration is in progress.

[YAML] Switched JavaScript engine from js2py to QuickJS (2.75.0)

Because js2py is no longer actively maintained, lacked Python 3.12+ support, and was affected by CVE-2024-28397, Beam YAML switched its JavaScript UDF execution engine to quickjs.

IO dependencies

[Java] Upgraded Apache Iceberg to 1.9.2 (2.68.0) and 1.10.0 (2.69.0)

Iceberg 1.7.0 dropped Java 8 support. After enabling individual Beam components to be compiled with different Java versions, IcebergIO was updated to build on top of Iceberg 1.9.2 in Beam 2.68.0. Later in Beam 2.69.0, it was upgraded to 1.10.0.

[Java] Upgraded HCatalogIO to Hive 4.0.1 (2.71.0)

HCatalog 4.0 dropped Java 8 support. Beam moved from HCatalog 3.1.3 to 4.0.1 in Beam 2.71.0, requiring Java 11.

[Java] Elasticsearch 9 Support (2.71.0)

[Java] Upgraded MongoDB Java driver from 3.x to 5.5.0 (2.68.0)

[Java] Migrated ClickHouseIO to ClickHouse Java Client v2 (2.72.0)

Runners

Apache Flink 2.x Runner Support (2.0 since 2.72.0, 2.1 and 2.2 since 2.75.0) and dropped Flink 1.17 and 1.18 runner support (2.75.0)

Flink 2.0 dropped the legacy DataSet API. Batch pipelines now run on the Flink DataStream API.

Apache Spark 4 Runner Support (2.74.0)

Spark 3.5 (and 4.0) deprecated the DStream API and improved the Dataset (Structured Streaming) API. Spark 4 streaming support is in progress.

Programming language interface modernization

Java

JUnit 5 support (2.69.0)

Previously, TestPipeline relied on JUnit 4’s TestRule (@Rule). Beam now provides a dedicated org.apache.beam:beam-sdks-java-testing-junit module featuring TestPipelineExtension, enabling JUnit 5 tests to inject TestPipeline via @ExtendWith (or configure custom options with @RegisterExtension) while maintaining backward compatibility with JUnit 4:

import org.apache.beam.sdk.testing.TestPipelineExtension;
...

@ExtendWith(TestPipelineExtension.class)
class MyPipelineTest {

 @Test
 void testPipeline(TestPipeline pipeline) {
 PCollection<String> output = pipeline.apply("Create", Create.of("hello", "world"));

 PAssert.that(output).containsInAnyOrder("hello", "world");
 pipeline.run();
 }
}

DoFn OutputReceiver fluent OutputBuilder API for extended metadata (2.69.0)

To support extensible per-element metadata (such as Change Data Capture operations, OpenTelemetry trace propagation, and pipeline drain indicators) without introducing combinatorial outputWith* method overloads on OutputReceiver and ProcessContext, DoFn.OutputReceiver now provides a fluent OutputBuilder API (receiver.builder(element)...output()). See the Beam Element Extended Metadata design doc for details.

Python

Dataclass-backed Beam portable Row (2.73.0)

Python 3.7 (PEP 557) introduced dataclass and recommended it over NamedTuple, while the Beam Python SDK previously only used NamedTuple for Beam Rows under the Beam portable schema framework.

Beam Row also had limited support for custom types: an arbitrary object type could be assigned to a Beam Row and work within the Python SDK, yet break across language boundaries.

Incremental improvements have been made to address these limitations:

Support dataclass for Beam Row (2.73.0).
Preserving registered NamedTuple and dataclass types with register_row and field type inference (2.74.0, 2.75.0).
Support for Python user types in Beam SQL (2.74.0).

Using Beam SQL with the Python SDK now supports rows containing arbitrary Python user types, as well as Python built-in types:

class Arbitrary:
 def __init__(self, obj):
 self.obj = obj

 def __eq__(self, other):
 return self.obj == other.obj

with Pipeline() as p:
 out = (
 p | beam.Create([
 UserRow(1, Arbitrary(1.0), 1 + 2.5j),
 UserRow(1, Arbitrary("abc"), -1j),
 ])
 | SqlTransform("SELECT arb, complex FROM PCOLLECTION"))
# result: [(Arbitrary(1.0), 1 + 2.5j), (Arbitrary("abc"), -1j)]

Made Beartype the default fallback runtime type checker (2.74.0)

Beam’s custom typehint system previously fell back to Any for newer standard Python typing constructs. Integrating beartype as the default fallback enables fast runtime type checking for modern Python type annotations (which can be disabled if needed via --disable_beartype).

Enhanced typehint support

for tagged output type hints (2.72.0)
for exception handling (2.73.0, 2.74.0)

[Python] Split optional dependencies (2.69.0, 2.70.0)

Feature- and connector-specific dependencies (dill, hadoop, redis, interactive, tfrecord, yaml) have been split out of the core apache-beam package into optional extras (e.g., pip install apache-beam[gcp,interactive,yaml,redis,hadoop,tfrecord]), reducing the default installation footprint and minimizing transitive dependency conflicts.

Deprecations

Sunsets

[Runner] Removed Apache Samza runner support (2.74.0)

Apache Samza has been in a “dormant” state. As part of the Java 8 sunset, Samza runner support has been removed from Beam.

[IO] Removed Pub/Sub Lite IO support (2.72.0)

GCP Pub/Sub Lite is no longer available for new customers after September 24, 2024. In line with this, Pub/Sub Lite IO has been sunset from Beam.

Deprecations

Since the community has decided to keep the major version number for now and avoid deliberate breaking changes, the following components are marked as deprecated while remaining available.

[Java] Deprecated Beam Euphoria API (2.69.0)

[Java] Removed deprecated Hadoop 2.x and 3.2.x versions from IcebergIO (2.69.0)

[Python] Deprecated native Python SpannerIO in favor of cross-language wrapper (2.68.0)

[Runner] Deprecated Twister2 runner support (2.68.0)

UnboundedSource and the Watch Transform in the Apache Beam Python SDK

Thu, 10 Sep 2026 00:00:00 +1000

The Apache Beam Python SDK now has an UnboundedSource API for writing custom unbounded sources and a Watch transform for repeatedly polling an input that keeps growing. I built both during my Google Summer of Code 2026 project with Apache Beam, mentored by Yi Hu.

This post describes both APIs as of Beam 2.77.0.

Motivation

Writing a connector for a message broker or database change feed means deciding how to read records, save a position, and resume after a failure. Python already supported custom streaming reads through a splittable DoFn (SDF). Using one also meant learning how to represent work as a restriction, hand unfinished work back to the runner, and report progress through a watermark estimator. UnboundedSource wraps that machinery in a reader API so source authors can focus on their connector’s reading and checkpoint logic.

Polling a growing input raises a related problem: how to remember which results have already been emitted. Python’s fileio.MatchContinuously could poll for new files, but its deduplication state grew with the number of matched paths. Watch makes this polling logic reusable for other inputs, such as an API that lists newly available records. Its opt-in timestamp_cursor mode lets old deduplication history expire when the input’s event times keep advancing.

The UnboundedSource API

The first public Python UnboundedSource API addresses a long-standing gap between the Java and Python SDKs. Source authors implement UnboundedSource, UnboundedReader, and CheckpointMark, then read the source with beam.io.Read(MySource()).

The reader exposes methods such as start(), advance(), get_current(), get_current_timestamp(), and get_checkpoint_mark(). Reading must not block: returning False from start() or advance() means that no record is available now, and the reader can resume when more data arrives. The reader also reports an event-time watermark through get_watermark(), which Beam uses to track progress and determine when windows can close. A watermark of MAX_TIMESTAMP signals that the source has permanently finished.

The SDK runs the reader through an SDF. The wrapper saves the reader’s checkpoint with the unfinished work and reports its watermark to the runner. This lets the same source implementation run on DirectRunner, Prism, Flink, and Dataflow. Sources can split their work at pipeline startup; an active read is not subdivided further.

The wrapper uses bundle finalization to invoke CheckpointMark.finalize_checkpoint after the runner has durably committed the output. A message-queue source can use this hook to acknowledge consumed messages. Finalization is best effort: a mark may never be finalized, and retries can produce marks covering overlapping records. The hook must therefore be idempotent. Readers can also be reused across resumed bundles on the same worker, with idle readers evicted from a bounded cache, reducing the need to reopen connections.

Mentor review led me to limit how many records a reader can emit and how long it can run before yielding. The wrapper checks these limits between reads. A busy source needs to yield regularly so the runner can commit its progress and finalize checkpoints. The Python I/O connector guide includes an example source and explains the API’s lifecycle.

The Watch transform

The Python Watch transform ports Java’s polling transform. For each input element, it calls a user-supplied poll function, emits newly discovered outputs, and saves progress between rounds. Polling stops when the poll reports completion or a termination condition fires. The API includes PollFn, PollResult, and the never() and after_total_of() termination conditions.

A single SDF manages each input’s polling, duplicate suppression, output, waiting, and termination. For example, a poll can repeatedly list files under a prefix while Watch remembers which results it has already emitted. Keeping this lifecycle together also lets the transform save its deduplication state with its progress.

An output’s identity is the hash of its encoded key. The key defaults to the output itself, and output_key_fn can select another identity. Watch requires a deterministic key coder so equal keys produce the same fingerprint across workers and after a restart. A coder with no deterministic form is rejected when the pipeline is built.

The default deduplication mode retains a hash for every distinct output key, so its history grows throughout a long-running watch. This also allows the transform to recognize an item seen much earlier. The opt-in timestamp_cursor mode addresses this state-growth problem by letting history expire as event time advances.

The cursor records the greatest emitted event time. Outputs more than allowed_lateness behind it are skipped, including previously unseen ones, and hashes older than that threshold can be discarded. This suits inputs arriving in roughly non-decreasing event time. Increasing allowed_lateness accommodates older arrivals while retaining more history. The cursor itself is a single timestamp; the retained hashes depend on the keys within that time range. In cursor mode, an item must keep its original event time across polls; assigning it a new timestamp on every poll can cause it to be emitted again after its hash expires.

Refactoring MatchContinuously onto Watch replaced its per-file state entries with the Watch restriction, so continuous file matching can use cursor mode and stop accumulating an entry for every file it has ever matched. The existing implementation remains for users who disable duplicate suppression. The cursor design was also ported back to Java.

Validation across runners

I tested both transforms on DirectRunner, Prism, Flink, and Dataflow. The runs covered pause and resume behavior, acknowledgments, watermarks, and polling. The UnboundedSource wrapper passed five end-to-end tests submitted as Dataflow streaming jobs. For MatchContinuously on Flink, testing included killing a worker during a run and restoring from a checkpoint. Prism tests added files while a watch was running and checked that both deduplication modes emitted them once and terminated on time.

These runs exposed issues beyond the SDK implementations:

Flink accumulated state entries when an SDF saved unfinished work. Reusing a state entry addressed the growth.
Prism could leave downstream records unprocessed when a source paused and resumed without advancing its watermark. Consumers with new data are now scheduled in that case.
Portable Spark batch gained support for retaining and resuming unfinished SDF work.

The work also produced a local Flink contributor guide, documenting the cluster setup used to reproduce and investigate streaming behavior.

Benchmarks

The local benchmarks measured UnboundedSource throughput and checkpoint cadence, and Watch deduplication overhead as the polled set grew.

For UnboundedSource, an in-memory source supplied one million records to isolate the wrapper’s overhead from external I/O. On Prism, a cap of 1,000 records per invocation produced 1,001 self-checkpoints and about 34,000 records per second. Raising the cap to 10,000 reduced the self-checkpoint count to 101 and reached about 44,000 records per second. A cap of 100,000 reduced the count to 11, with throughput still around 44,000 records per second. Throughput was measured from the first record to the last, excluding runner startup.

The Watch benchmark repeatedly listed a set that gained 2,000 items per round for 100 rounds. Each item retained its original event time. Both modes emitted all 200,000 items once. Cursor mode reduced total time from 111 to 24 seconds on DirectRunner and from 59 to 15 seconds on Prism. These single-machine experiments show how checkpoint frequency and growing deduplication history affect the transforms; distributed benchmarks remain future work.

Remaining work

Both Python APIs remain experimental, and Spark streaming SDF support is still open. The full project report includes the contribution list, documentation, validation details, and benchmark methodology.

Thank you to my mentor, Yi Hu, and the Apache Beam community for their guidance and reviews throughout Google Summer of Code 2026.

Apache Beam 2.76.0

Mon, 31 Aug 2026 14:00:00 -0500

We are happy to present the new 2.76.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.76.0, check out the detailed release notes.

Highlights

Added a full Iceberg batch and streaming changelog source (CDC) (#38831)
(Java) Added per-element OpenTelemetry trace propagation across stages in the Dataflow Streaming Runner. Enable it with --experiments=enable_otel_defaults,element_metadata_supported,disable_portable_worker. Cloud Trace incurs additional cost. (#33176)
(Java) Added OpenTelemetry header propagation support for both reads and writes in KafkaIO and PubSubIO. (#33176)
(Java) Added OpenTelemetry tracing support for SpannerIO change streams (#33176)
(Python) JmsIO (IBM MQ, ActiveMQ, and other providers) is now supported in Python via cross-language (#30716).

I/Os

Upgraded Iceberg dependency to 1.11.0 (Java) (#38925).
Add ArrowFlight IO (Java) (#20116).
Added a Delta Lake batch changelog source (CDC) (#39492)

New Features / Improvements

Added GroupIntoBatches transform and the standard beam:coder:sharded_key:v1 coder to the Go SDK, along with beam.Coder.IsDeterministic, beam.PCollection.WindowingStrategy, and coder.RegisterDeterministicCoder for opt-in deterministic custom coders (Go) (#19868).
TriggerStateMachineRunner changes from BitSetCoder to SentinelBitSetCoder to encode finished bitset. SentinelBitSetCoder and BitSetCoder are state compatible. Both coders can decode encoded bytes from the other coder (#38139).
(Python) Removed the envoy-data-plane (and transitive betterproto) dependency; EnvoyRateLimiter now uses a small vendored protobuf definition instead, resolving dependency conflicts for downstream projects (#37854).
(Java) Supported acknowledge mode for JmsIO (#39253).
(Python) Staged files directory is now automatically added to sys.path on the Python SDK worker at startup. This makes Python files provided via the ‘–files_to_stage’ pipeline option importable in the pipeline code and makes it easier to initialize Python SDK harness at startup via the --beam_plugins pipeline option. For more information, see the Staging Individual Files section of the dependency management docs. This behavior can be disabled by passing the ‘–experiments=no_staged_dir_in_sys_path’ pipeline option (#39431).
(Python) Added equal_to_approx, an assert_that matcher that compares numeric pipeline outputs with a configurable tolerance (#18028).
(Python) Timestamp now supports variable subsecond precision, up to nanoseconds. The portable beam:logical_type:timestamp:v1 logical type now maps to Python’s Timestamp (#39344).
(Python) Added UnboundedSource, an interface for reading an infinite stream of records with checkpointing, watermark reporting, and bundle finalization. Read one with beam.io.Read (#19137).
(Python) Added Watch, a transform that polls a growing set of outputs for each input element, deduplicates outputs across poll rounds, and stops per a user-supplied termination condition (#21521).
(Python) Added support to analyze core dumps created after python worker segmentation faults with pystack (or gdb if installed) using the --profiler_agent=coredump pipeline option. (#39484).

Breaking Changes

(Python) Removed google-perftools from the SDK container images. Users who wish to use --profiler_agent=tcmalloc should install google-perftools APT package in their custom container images separately (#39323).
[IcebergIO] Reading a timestamptz column will now return a Timestamp.MICROS Beam logical type to preserve microseconds (the old Beam Schema.FieldType#DATETIME primitive type truncates past milliseconds). This may break the following use cases when a timestamptz column is present:
- Existing streaming read pipelines.
- Managed Iceberg batch reads when upgraded from an older SDK.
- Python reads.
Use pipeline option --updateCompatibilityVersion=2.75.0 (or any older version) to keep the old behavior (#39344).
DoFn.process returning a str, bytes, or dict (instead of an iterable wrapping one) now raises a TypeError rather than silently iterating per-character/byte/key (Python) (#18712).
(Java) Added DRAINING and DRAINED states to PipelineResult, including runner state mappings and Dataflow update handling (#39020).
(Java) IcebergIO and projects that use it must now be built with Java 17 or later as a result of Iceberg 1.11.0 upgrade (#38925).

Bugfixes

Fixed unresolved runtime ValueProvider options being stringified in Python Dataflow Flex Templates (#39499).
Fixed unbounded checkpoint state growth for splittable DoFns that self-checkpoint on the portable Flink runner (Java) (#27648).
Improved Java pipeline performance by avoiding repeated DoFn type descriptor resolution when creating cached invokers (#39309).
(Python) Fixed a memory leak in Python SDK caused by storing exceptions with potentially large stack frames in a cache (#39406).

Known Issues

(Java) Projects using the Flink runner with Flink 2.1 or later alongside libraries requiring org.lz4:lz4-java (e.g., Kafka clients) may encounter a Gradle capability conflict, because Flink 2.1+ ships at.yawk.lz4:lz4-java which declares the same capability. To resolve, add a capabilitiesResolution rule to your build.gradle that selects at.yawk.lz4:lz4-java (#38947).

According to git shortlog, the following people contributed to the 2.76.0 release. Thank you to all contributors!

ADITYA RAJ, Abdelrahman Ibrahim, Ahmed Abualsaud, Alexander Kolb, Amar3tto, Andrew Crites, Arun Pandian, Aryankn29, Atharva Moroney, Avi Kondareddy, Bruno Volpato, Chamikara Jayalath, Chris Qiu, Claire McGinty, Danny McCormick, Derrick Williams, Elia Liu, Florian TREHAUT, Guflly, HansMarcus01, Ian Liao, Ivy Xu, Jack McCluskey, KRITI MITTAL, Kenneth Knowles, Lalit Yadav, Manvith Panyam, Minh Vu, Nikita Grover, PRADDZY, Peter Tran, Radosław Stankiewicz, Ryan Wigglesworth, Shahar Epstein, Shunping Huang, SreeramaYeshwanthGowd, Steven van Rossum, Tarun Annapareddy, Tejas Iyer, Tobias Kaymak, Tomasz Wojdat, Utkarsh Parekh, Venkata Bharath Malapati, Vitaly Terentyev, Yi Hu, ZIHAN DAI, aibrahiim, akshayjadiyanv, atognolas, claudevdm, janaom, jayjayakumar, raman118, shunping, tvalentyn

Beam Summit 2026 | Interview with Raj Katakam, Intuit Credit Karma

Wed, 22 Jul 2026 15:00:00 -0500

Beam Summit 2026 | Interview with Raj Katakam, Intuit Credit Karma

Apache Beam and Google Cloud Dataflow are usually known for ETL and batch/streaming processing. Intuit Credit Karma said, why stop there, and built an entire ML platform on top of them.

Here is my interview with Raj Katakam, Staff Machine Learning Engineer at Intuit Credit Karma. We talked about how they built a unified ML platform on Apache Beam and Dataflow to serve 140 million users, why being active in the Apache Beam community matters to him, and what advice he would give anyone just getting started with Beam and ML.

Jana: Intuit Credit Karma’s tech stack looks like a dream! You are on Google Cloud and using services like Dataflow, BigQuery, and Managed Airflow (previously, Cloud Composer). What were the main reasons for choosing this stack, and what trade-offs did you have to consider when selecting GCP and Apache Beam/Dataflow?

Raj: The selection of Google Cloud, utilizing BigQuery, Dataflow, and Managed Airflow, was a strategic decision to standardize the ML lifecycle and enable high-velocity development at scale. By leveraging these technologies, we established a unified architectural standard that allows for elastic scalability and reliability across petabyte-scale workloads. This approach ensures that the entire ML lifecycle, from initial experimentation to global production execution, is governed by a consistent framework, providing the necessary infrastructure to manage complex data movements and distributed computing requirements without sacrificing architectural integrity.

Our goal was to create a unified abstraction layer that could bridge various data processing tools like Spark and BigQuery with ML libraries such as scikit-learn and TensorFlow. GCP provided the deep, native integrations we required to smoothly transition from local prototypes to massive, distributed cloud execution.

Talking about trade-offs, the biggest hurdle was simplifying the inherent complexity of a multi-framework environment. Even with powerful connectors for GCS and BigQuery, we still had to develop Vega (Intuit Credit Karma’s internal ML platform) to shield our data scientists from manual orchestration and resource management, allowing them to remain focused on a clean, unified Python API.

Jana: Dataflow is widely known for ETL and batch/streaming processing, but you took it a step further by using it for ML. You built Vega, a full ML explainability platform on Apache Beam and Dataflow. What motivated that choice, and what advantages did these technologies bring to the project?

Raj: We utilized Dataflow as the distributed execution engine to power the broader Vega ecosystem, which serves as our comprehensive ML platform. It is important to emphasize that explainability is a foundational pillar of this ecosystem, integrated directly into the core architecture rather than treated as a standalone task. By building on Apache Beam, we ensure that every model, whether focused on feature engineering, scoring, or interpretability, benefits from a unified processing model. This integration allows us to maintain rigorous standards for model transparency and governance as a native component of our large-scale distributed ML operations.

The main motivation was to eliminate the experimentation-to-production gap. Traditional platforms forced data scientists to manually rewrite code for production frameworks, but by using Dataflow and Apache Beam as the core engine, we were able to support both batch and streaming processing alongside ML scoring within a single workflow.

The primary advantage that came out of this was the unified Python API. It allowed data scientists to build complex, end-to-end data and ML pipelines that transition seamlessly from local development on sampled data to cloud-scale production execution, without architectural changes or manual re-engineering.

Jana: Looking back at building Vega, what were the biggest challenges you faced, and what would you do differently today?

Raj: The primary challenges centered on managing the inherent complexities of distributed systems and maintaining architectural coherence across a diverse ML lifecycle. Operating at the scale of over 140 million members requires solving for high-concurrency data access, managing complex dependency graphs in distributed environments, and ensuring that performance remains consistent as workloads transition from sampled data to massive production clusters. Balancing the need for developer flexibility with the strict latency and reliability constraints of a globally distributed platform represents the most significant architectural hurdle in modern ML engineering.

Reflecting on the evolution of Vega, the biggest challenge was navigating the trade-offs of building critical infrastructure while scaling to support massive business growth with limited resources. We prioritized speed-to-market and immediate utility to enable the business, which was necessary but created technical debt in terms of documentation and onboarding. If I were starting today, I would prioritize earlier investments in developer tooling and standardized templates. We are now shifting our focus from that initial fast-and-lean build phase to long-term sustainability, refining the developer experience, automating maintenance even further, and broadening support for modern ML paradigms. This ensures that the platform remains a scalable, durable asset as it evolves to meet the next generation of business needs.

Jana: Are there any features or improvements you would love to see in Apache Beam or Dataflow in the future?

Raj: While Vega has successfully leveraged Beam for our core workflows, there are a few areas where future improvements could make a real difference. One would be enhanced data engineering primitives. We would value deeper native support for large-scale data manipulation, specifically generalized tiling operations for distributed datasets and built-in capabilities for saving and managing partial processing states. This would significantly reduce the complexity of checkpointing and state recovery in our pipelines.

The other area is real-time inference. While we have strong batch capabilities, we need more streamlined primitives in Beam to better integrate our data pipelines with real-time serving infrastructure. Reducing the complexity of the hand-off between processing and prediction would be a major win for our personalization engines.

Jana: You are a Staff Machine Learning Engineer at Intuit Credit Karma — could you tell us a bit about your background and journey to this role? And for readers who are interested in pursuing a career in ML, what advice would you give them?

Raj: As a Staff Machine Learning Engineer, my career has been defined by the practice of platform-scale engineering. I have focused on architecting ecosystems that provide high-velocity developer experiences, allowing engineering teams to move from conceptual design to production-scale deployment within a unified framework. By treating the ML lifecycle as a first-class engineering problem, we build systems that automate the complexities of infrastructure management, enabling data scientists to focus on model innovation while the platform handles the rigorous demands of distributed execution and governance.

My advice to anyone pursuing a career in ML would be to focus on first-principles thinking. Don’t just learn a specific library, understand the underlying data infrastructure, including data movement, latency constraints, and caching. The most impactful engineers I have worked with, including those I have collaborated with at Google and beyond, are the ones who can reason across domains, from low-level feature serving latency to high-level platform architecture.

Jana: You are an active member of the Apache Beam community. This is your third Beam Summit. How important do you think community involvement is for engineers, and how has it shaped your career?

Raj: Community involvement is critical. Participating in initiatives like the Apache Beam Summit has allowed me to stay at the frontier of distributed computing and directly influence the evolution of the tools we depend on. It transforms engineering from a siloed task into a collaborative effort; seeing how other engineers solve (or struggle with) similar problems helps refine your own architectural decisions and often provides the reference architectures that we would otherwise have taken years to develop independently.

Jana: If you could give one piece of advice to someone just starting with Apache Beam, what would it be?

Raj: Focus on the abstraction. Do not treat Beam as just another processing framework; treat it as the “universal language” for your data pipeline. Start by leveraging the high-level transforms and built-in IO connectors rather than trying to optimize low-level custom code immediately. Understanding how to structure your pipeline to take advantage of Beam’s windowing and triggering capabilities is what will eventually separate a good engineer from a great one when you hit the scale of millions of users.

Jana: Thank you so much, Raj, for sharing these insights!

Raj: Thank you for having me, Jana, this was a great conversation!

🌐 To learn more about Intuit Credit Karma, visit their website or check out their LinkedIn. You can also connect with Raj directly on LinkedIn.

📌 Raj and Pallav Anand presented at Beam Summit 2026. Check out their talk, ‘Beyond the Black Box: How Intuit Credit Karma Runs ML Explainability for 140M Members with Beam’, and explore the full program at beamsummit.org.

📺 Session recording will be available on the Apache Beam YouTube channel.

📰 Missed the Beam Summit 2026? Read my recap: Beam Summit 2026: Apache Beam Just Got Even More Interesting 🐝

– Jana Polianskaja

Apache Beam 2.75.0

Wed, 08 Jul 2026 14:00:00 -0500

We are happy to present the new 2.75.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.75.0, check out the detailed release notes.

Highlights

Python SDK now supports memory profiling with Memray (#38853).
(Python) Added Qdrant VectorDatabaseWriteConfig implementation (#38141).

I/Os

Support for reading from Delta Lake added (Java) (#38551).
ClickHouseIO: support writing DateTime64(precision[, 'timezone']) columns with sub-second precision (Java) (#38466).
Upgraded IO Expansion Service to Java 17 (#38974).

New Features / Improvements

Dataflow Runner v2 has been renamed to Dataflow Portable Runner. Please refer to Dataflow public documentation on when to enable Portable Runner.(#39000).
(Java) Enabled state tag encoding v2 by default for new Dataflow Streaming Engine jobs. It can be disabled by passing --experiments=disable_streaming_engine_state_tag_encoding_v2 or --updateCompatibilityVersion=2.74.0 pipeline option. Note that the tag encoding version cannot change during a job update. Jobs using tag encoding v2 (enabled by default for new jobs on 2.75.0+) cannot be downgraded to Beam versions prior to 2.73.0, as only versions 2.73.0 and later support tag encoding v2. (#38705).
(Python) Added instrumentation to support off-the-shelf profiling agents when launching Python SDK Harness (#38853).
(Java) Added support to the FnApi Data stream protocol allowing runners to isolate bundles slowly processing input from other bundles. (#39001).
(YAML) Switched js2py library to Quickjs (#38473).
(YAML) Added HuggingFaceModelHandler for YAML usage (#38696).
(YAML) Added WriteToMongoDB transform (#38376).
(YAML) Added WriteToDatadog transform (#38362).
(Java) Flink 2.1 and 2.2 support is added (#38947) (#38978); Flink 1.17 and 1.18 support is dropped.
(Python) MqttIO is now supported in Python via cross-language (#21060).

Breaking Changes

(Python) Typehints of dataclass fields are honored during type inferences. To restore the behavior of fallback-to-any, use pipeline option --exclude_infer_dataclass_field_type (#38797). However fixing forward is recommended.

Bugfixes

Fixed GCS filesystem glob matching to correctly handle / in object names and support ** for recursive matching (Go) (#38059).
Fixed BigQueryEnrichmentHandler batch mode dropping earlier requests when multiple requests share the same enrichment key (Python) (#38035).
Fixed IcebergIO writing manifest column bounds padded with trailing 0x00 bytes, which broke equality predicate pushdown in some query engines (Java) (#38580).

Known Issues

(Java) Projects using the Flink runner with Flink 2.1 or later alongside libraries requiring org.lz4:lz4-java (e.g., Kafka clients) may encounter a Gradle capability conflict, because Flink 2.1+ ships at.yawk.lz4:lz4-java which declares the same capability. To resolve, add a capabilitiesResolution rule to your build.gradle that selects at.yawk.lz4:lz4-java (#38947).

According to git shortlog, the following people contributed to the 2.75.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Akshat, Andrew Crites, Andrew Kabas, Anurag Pappula, Arpit Jain, Arun Pandian, Atharv, Chamikara Jayalath, Danny McCormick, Deji Ibrahim, Derrick Williams, Drew Stevens, Durgaprasad M L, Elia Liu, Enzo Maruffa Moreira, Ganesh Sivakumar, Goutam Adwant, HansMarcus01, Jack McCluskey, Joe Santos, Kenneth Knowles, Lalit Yadav, Liam Miller-Cushon, Maciej Szwaja, Manan Mangal, Michael Gruschke, Nikita Grover, Radek Stankiewicz, Radosław Stankiewicz, Reuven Lax, RuiLong J., Sachin Ranjalkar, Sagnik Ghosh, Sam Whittle, Shunping Huang, Subramanya V, Tarun Annapareddy, Tobias Kaymak, TongruiLi, Valentyn Tymofieiev, Vitaly Terentyev, XQ Hu, Yi Hu, aaaZayne, claudevdm, ddebowczyk92, innuendo, kellen, parveensania, tejasiyer-dev

Apache Beam 2.74.0

Tue, 02 Jun 2026 14:00:00 -0500

We are happy to present the new 2.74.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.74.0, check out the detailed release notes.

Highlights

Spark 4 runner support for Java SDK (#38255).

I/Os

IcebergIO: support declaring a table’s sort order on dynamic table creation via the new sort_fields config (#38269).
IcebergIO: support writing with hash distribution mode, and with autosharding (#38061).

New Features / Improvements

Capability introduces an indicator for aggregations and timers firing during a pipeline drain, allowing users and sinks to recognize and appropriately handle potentially incomplete or partial data (#36884).
Added support for setting disk provisioned IOPS and throughput in Dataflow runner via --diskProvisionedIops and --diskProvisionedThroughputMibps pipeline options (Java/Go/Python) (#38349).
TriggerStateMachineRunner changes from BitSetCoder to SentinelBitSetCoder to encode finished bitset. SentinelBitSetCoder and BitSetCoder are state compatible. Both coders can decode encoded bytes from the other coder (#38139).
(Python) Added type alias for with_exception_handling to be used for typehints. (#38173).
(Java) BatchElements transform for Java SDK (#38369)
Added plugin mechanism to support different Lineage implementations (Java) (#36790).
(Python) Supported Python user type in Beam SQL. For example, SQL statements like SELECT some_field from PCOLLECTION can now operate a PCollection of Beam Row containing pickable Python user type (#20738).
(Python) Introduced beam.coders.registry.register_row as preferred API to register a named tuple or dataclass with a Beam Row. At pipelne runtime, the original type associated with the registered row are preserved across the serialization boundary (#38108).

Breaking Changes

(Python) Made Beartype the default fallback type checking tool. This can be disabled with the --disable_beartype pipeline option. (#38275)

Deprecations

Dropped Java 8 support (#31678).
Removed Samza Runner support (#35448).

Bugfixes

Fixed BigQueryEnrichmentHandler batch mode dropping earlier requests when multiple requests share the same enrichment key (Python) (#38035).
Added max_batch_duration_secs passthrough support in Python Enrichment BigQuery and CloudSQL handlers so batching duration can be forwarded to BatchElements (#38243).

According to git shortlog, the following people contributed to the 2.74.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Andrew Crites, Andrew Kabas, Arran Cudbard-Bell, Arun Pandian, Asish Kumar, Bentsi Leviav, Blake Jones, Bruno Volpato, Chris Jordan, Danny McCormick, Deji Ibrahim, Derrick Williams, Elia LIU, Ganesh Sivakumar, Jack McCluskey, Kenneth Knowles, Lalit Yadav, M Junaid Shaukat, Matej Aleksandrov, Prabhnoor Singh, Radek Stankiewicz, Radosław Stankiewicz, Reuven Lax, RuiLong J., Sam Whittle, Shunping Huang, Subramanya V, Tarun Annapareddy, Tobias Kaymak, TongruiLi, Valentyn Tymofieiev, Vitaly Terentyev, XQ Hu, Yi Hu, ZIHAN DAI, apanich, bambadiouf1, chenxuesdu, claudevdm, harshadkhetpal, johnjcasey, parveensania, tianz101

Apache Beam 2.73.0

Wed, 29 Apr 2026 09:00:00 -0700

We are happy to present the new 2.73.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.73.0, check out the detailed release notes.

Highlights

I/Os

DebeziumIO (Java): added OffsetRetainer interface and FileSystemOffsetRetainer implementation to persist and restore CDC offsets across pipeline restarts, and exposed withStartOffset / withOffsetRetainer on DebeziumIO.Read and the cross-language ReadBuilder (#28248).

New Features / Improvements

(Python) Added BigQuery CDC streaming source (#37724)
Added ADKAgentModelHandler for running Google Agent Development Kit (ADK) agents (Python) (#37917).
(Python) Added exception chaining to preserve error context in CloudSQLEnrichmentHandler, processes utilities, and core transforms (#37422).
(Python) Added a pipeline option --experiments=pip_no_build_isolation to disable build isolation when installing dependencies in the runtime environment (#37331).
(Go) Added OrderedListState support to the Go SDK stateful DoFn API (#37629).
Added support for large pipeline options via a file (Python) (#37370).
Supported infer schema from dataclass (Python) (#22085). Default coder for typehint-ed (or set with_output_type) for non-frozen dataclasses changed to RowCoder. To preserve the old behavior (fast primitive coder), explicitly register the type with FastPrimitiveCoder.
Updates minimum Go version to 1.26.1 (#37897).
(Python) Added image embedding support in apache_beam.ml.rag package (#37628).
(Python) Added support for Python version 3.14 (#37247).

Breaking Changes

The Python SDK container’s boot.go now passes pipeline options through a file instead of the PIPELINE_OPTIONS environment variable. If a user pairs a new Python SDK container with an older SDK version (which does not support the file-based approach), the pipeline options will not be recognized and the pipeline will fail. Users must ensure their SDK and container versions are synchronized (#37370).
Python DoFn.with_exception_handling now respects user DoFn typehints. This can break update compatibility if coders change. It can also break pipeline compilation if existing typehints are incorrect. To update safely sepcify the pipeline option --update_compatibility_version=2.72.0. To fix typehints replace any incorrect typehints that were previously ignored (#37590)

Bugfixes

Fixed ProcessManager not reaping child processes, causing zombie process accumulation on long-running Flink deployments (Java) (#37930).

Security Fixes

Fixed CVE-2023-46604 (CVSS 10.0) and CVE-2022-41678 by upgrading ActiveMQ from 5.14.5 to 5.19.2 (Java) (#37943).
Fixed CVE-2024-1597, CVE-2022-31197, and CVE-2022-21724 by upgrading PostgreSQL JDBC Driver from 42.2.16 to 42.6.2 (Java) (#37942).

List of Contributors

According to git shortlog, the following people contributed to the 2.73.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Alex Malao, Alexander Nieuwenhuijse, Andres Tiko, Andrew Crites, Arun Pandian, Bentsi Leviav, Bruno Volpato, Chamikara Jayalath, Chandra Kiran Bolla, Danny McCormick, Deji Ibrahim, Derrick Williams, Elia LIU, Esmelealem, Hannes Gustafsson, Jack McCluskey, Joey Tran, Kenneth Knowles, M Junaid Shaukat, Mansi Singh, Matej Aleksandrov, Mathijs Deelen, Mattie Fu, Praneet Nadella, Radek Stankiewicz, Radosław Stankiewicz, Reuven Lax, RuiLong J., S. Veyrié, Sakthivel Subramanian, Sam Whittle, Shubham Thakur, Shunping Huang, Subramanya V, Tarun Annapareddy, Tobias Kaymak, Valentyn Tymofieiev, Vitaly Terentyev, XQ Hu, Yi Hu, ZIHAN DAI, claudevdm, kishorepola, parveensania

Apache Beam 2.72.0

Mon, 30 Mar 2026 09:00:00 -0700

We are happy to present the new 2.72.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.72.0, check out the detailed release notes.

Highlights

Flink 2.0 support (#36947).

I/Os

Add Datadog IO support (Java) (#37318).
Remove Pubsublite IO support, since service will be deprecated in March 2026. (#37375).
(Java) ClickHouse - migrating from the legacy JDBC driver (v0.6.3) to ClickHouse Java Client v2 (v0.9.6). See the class documentation for migration guide (#37610).
(Java) Upgraded GoogleAdsIO to use GoogleAdsIO API v23 (#37620).

New Features / Improvements

(Python) Added exception chaining to preserve error context in CloudSQLEnrichmentHandler, processes utilities, and core transforms (#37422).
(Python) Added a pipeline option --experiments=pip_no_build_isolation to disable build isolation when installing dependencies in the runtime environment (#37331).

Deprecations

(Python) Removed previously deprecated list_prefix method for filesystem interfaces (#37587).

Bugfixes

Fixed (Yaml) issue with validate compatible method (#37588).
Fixed (Yaml) issue with Create transform dealing with different type elements (#37585).

Security Fixes

Fixed CVE-2024-28397 by switching from js2py to pythonmonkey (Yaml) (#37560).

List of Contributors

According to git shortlog, the following people contributed to the 2.72.0 release. Thank you to all contributors!

Abdelrahman Ibrahim, Ahmed Abualsaud, Andrew Crites, Arun Pandian, Ben Feinstein, Bentsi Leviav, Celeste Zeng, Danny McCormick, Danny Mccormick, Derrick Williams, Elia LIU, Ganesh, Jack McCluskey, Kenneth Knowles, Labesse Kévin, M Junaid Shaukat, Mansi Singh, Mattie Fu, Nayan Mathur, Pablo Estrada, Pirzada Ahmad Faraz, Radek Stankiewicz, Radosław Stankiewicz, Robert Bradshaw, Rohan Sah, RuiLong J., Sakthivel Subramanian, Sam Whittle, Shaheer Amjad, Shunping Huang, Steven van Rossum, Tarun Annapareddy, Tobias Kaymak, Valentyn Tymofieiev, Vitaly Terentyev, Yi Hu, XQ Hu, ZIHAN DAI, apanich, chenxuesdu, claudevdm, franzonia137

Apache Beam 2.71.0

Tue, 13 Jan 2026 09:00:00 -0700

We are happy to present the new 2.71.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.71.0, check out the detailed release notes.

I/Os

(Java) Elasticsearch 9 Support (#36491).
(Java) Upgraded HCatalogIO to Hive 4.0.1 (#32189).

New Features / Improvements

Support configuring Firestore database on ReadFn transforms (Java) (#36904).
(Python) Inference args are now allowed in most model handlers, except where they are explicitly/intentionally disallowed (#37093).

Bugfixes

Fixed FirestoreV1 Beam connectors allow configuring inconsistent project/database IDs between RPC requests and routing headers #36895 (Java) (#36895).
Logical type and coder registry are saved for pipelines in the case of default pickler (#36271). This fixes a side effect of switching to cloudpickle as default pickler in Beam 2.65.0 (Python) (#35738).

Known Issues

For the most up to date list of known issues, see https://github.com/apache/beam/blob/master/CHANGES.md

List of Contributors

According to git shortlog, the following people contributed to the 2.71.0 release. Thank you to all contributors!

Ahmed Abualsaud, Andrew Crites, apanich, Arun, Arun Pandian, assaf127, Chamikara Jayalath, CherisPatelInfocusp, Cheskel Twersky, Claire McGinty, Claude, Danny Mccormick, Derrick Williams, Egbert van der Wal, Evan Galpin, Ganesh, hekk-kaori-maeda, Jack Dingilian, Jack McCluskey, JayajP, Jiang Zhu, Kenneth Knowles, liferoad, M Junaid Shaukat, Nayan Mathur, Noah Stapp, Paco Avila, Radek Stankiewicz, Radosław Stankiewicz, Robert Stupp, Sam Whittle, Shunping Huang, Steven van Rossum, Suvrat Acharya, Tarun Annapareddy, tvalentyn, Utkarsh Parekh, Vitaly Terentyev, Xiaochu Liu, Yala Huang Feng, Yi Hu, Yu Watanabe, zhan7236

Apache Beam

Apache Beam 2.77.0

Highlights

I/Os

New Features / Improvements

Breaking Changes

Bugfixes

Beam modernization at project 10-year mark

Summary

Package Upgrades

Core languages

[Java] Java 25 Support (2.69.0) and dropped Java 8 support (2.74.0)

[Python] Python 3.13 and 3.14 Support (2.69.0, 2.73.0) and removed Python 3.9 support (2.70.0)

[Go] Minimum Go version updated to 1.26 (2.69.0, 2.73.0)

[SQL] Upgraded Beam vendored Calcite from 1.28.0 to 1.40.0 for Beam SQL (2.68.0) and removed ZetaSQL support (2.68.0)

Core dependencies

[Java] Upgraded Avro to 1.12 (2.74.0)

[Java] Upgraded GCS connector (gcsio) to 3.x (2.74.0)

[Python] Protobuf 6.x Support (2.69.0)

[Python] Migrated default pickler to cloudpickle and made dill optional (2.68.0, 2.69.0)

[Python] Migrating GCP clients off google-apitools (2.72.0, 2.75.0)

[YAML] Switched JavaScript engine from js2py to QuickJS (2.75.0)

IO dependencies

[Java] Upgraded Apache Iceberg to 1.9.2 (2.68.0) and 1.10.0 (2.69.0)

[Java] Upgraded HCatalogIO to Hive 4.0.1 (2.71.0)

[Java] Elasticsearch 9 Support (2.71.0)

[Java] Upgraded MongoDB Java driver from 3.x to 5.5.0 (2.68.0)

[Java] Migrated ClickHouseIO to ClickHouse Java Client v2 (2.72.0)

Runners

Apache Flink 2.x Runner Support (2.0 since 2.72.0, 2.1 and 2.2 since 2.75.0) and dropped Flink 1.17 and 1.18 runner support (2.75.0)

Apache Spark 4 Runner Support (2.74.0)

Programming language interface modernization

Java

JUnit 5 support (2.69.0)

DoFn OutputReceiver fluent OutputBuilder API for extended metadata (2.69.0)

Python

Dataclass-backed Beam portable Row (2.73.0)

Made Beartype the default fallback runtime type checker (2.74.0)

Enhanced typehint support

[Python] Split optional dependencies (2.69.0, 2.70.0)

Deprecations

Sunsets

[Runner] Removed Apache Samza runner support (2.74.0)

[IO] Removed Pub/Sub Lite IO support (2.72.0)

Deprecations

[Java] Deprecated Beam Euphoria API (2.69.0)

[Java] Removed deprecated Hadoop 2.x and 3.2.x versions from IcebergIO (2.69.0)

[Python] Deprecated native Python SpannerIO in favor of cross-language wrapper (2.68.0)

[Runner] Deprecated Twister2 runner support (2.68.0)

UnboundedSource and the Watch Transform in the Apache Beam Python SDK

Motivation

The UnboundedSource API

The Watch transform

Validation across runners

Benchmarks

Remaining work

Apache Beam 2.76.0

Highlights

I/Os

New Features / Improvements

Breaking Changes

Bugfixes

Known Issues

Beam Summit 2026 | Interview with Raj Katakam, Intuit Credit Karma

Beam Summit 2026 | Interview with Raj Katakam, Intuit Credit Karma

Apache Beam 2.75.0

Highlights

I/Os

New Features / Improvements

Breaking Changes

Bugfixes

Known Issues

Apache Beam 2.74.0

Highlights

I/Os

New Features / Improvements

Breaking Changes

Deprecations

Bugfixes

Apache Beam 2.73.0

[Python] Migrating GCP clients off `google-apitools` (2.72.0, 2.75.0)