What is being computed?
| Google Cloud Dataflow | Prism Local Runner | Apache Flink | FlareDB | Apache Spark (RDD/DStream based) | Apache Spark Structured Streaming (Dataset based) | Hazelcast Jet | Kafka Streams (experimental, not released) | Twister2 | Python Direct FnRunner | |
|---|---|---|---|---|---|---|---|---|---|---|
| ParDo | Yes : fully supported Batch mode uses large bundle sizes. Streaming uses smaller bundle sizes. | Yes : fully supported Supported with per-element transformation. | Yes : fully supported ParDo itself, as per-element transformation with UDFs, is fully supported by Flink for both batch and streaming. | Yes : fully supported ParDo execution is supported | Yes : fully supported ParDo applies per-element transformations as Spark FlatMapFunction. | Partially : fully supported in batch mode ParDo applies per-element transformations as Spark FlatMapFunction. | Yes : fully supported | Yes : fully supported | Yes : fully supported Stateless ParDo runs in the SDK harness over the Fn API, including multiple outputs. | |
| GroupByKey | Yes : fully supported | Yes : fully supported | Yes : fully supported Uses Flink's keyBy for key grouping. When grouping by window in streaming (creating the panes) the Flink runner uses the Beam code. This guarantees support for all windowing and triggering mechanisms. | Yes : fully supported Uses Apache DataFusion to run GroupByKey operation. | Partially : fully supported in batch mode Using Spark's <tt>groupByKey</tt>. GroupByKey with multiple trigger firings in streaming mode is a work in progress. | Partially : fully supported in batch mode Using Spark's <tt>groupByKey</tt>. | Yes : fully supported | Yes : fully supported | Yes : fully supported Shuffles through a Kafka repartition topic keyed by the encoded Beam key. | |
| Flatten | Yes : fully supported | Yes : fully supported | Yes : fully supported | Yes : fully supported | Yes : fully supported | Partially : fully supported in batch mode Some corner cases like flatten on empty collections are not yet supported. | Yes : fully supported | Yes : fully supported | Yes : fully supported | |
| Combine | Yes : efficient execution | Yes : fully supported | Yes : fully supported Uses a combiner for pre-aggregation for batch and streaming. | Yes : fully supported CombinePerKey and CombineGlobally expand into combine sub-transforms that run in the SDK harness. | Yes : fully supported Using Spark's <tt>combineByKey</tt> and <tt>aggregate</tt> functions. | Partially : fully supported in batch mode Using Spark's <tt>Aggregator</tt> and agg function | Yes : fully supported Batch mode uses pre-aggregation | Yes : fully supported | Yes : fully supported Executed through the GroupByKey expansion; there is no lifted pre-combine yet. | |
| Composite Transforms | Partially : supported via inlining Currently composite transformations are inlined during execution. The structure is later recreated from the names, but other transform level information (if added to the model) will be lost. | Yes : supported via inlining | Partially : supported via inlining | Partially : supported via inlining Composite transforms are expanded and inlined by the SDK before the pipeline reaches the runner. | Partially : supported via inlining | Partially : supported via inlining only in batch mode | Partially : supported via inlining | Partially : supported via inlining | Yes : fully supported | |
| Side Inputs | Yes : some size restrictions in streaming Batch mode supports a distributed implementation, but streaming mode may force some size restrictions. Neither mode is able to push lookups directly up into key-based sources. | Yes : fully supported | Yes : some size restrictions in streaming Batch mode supports a distributed implementation, but streaming mode may force some size restrictions. Neither mode is able to push lookups directly up into key-based sources. | No : not implemented Side inputs are modeled in fusion and watermarking, but side-input data is not delivered to the SDK harness (only bag user state is served). | Yes : fully supported Using Spark's broadcast variables. In streaming mode, side inputs may update but only between micro-batches. | Partially : fully supported in batch mode Using Spark's broadcast variables. | Partially : with restrictions Supported only when the side input source is bounded and windowing uses global window | Yes : fully supported | No : not implemented Stages run without a side input handler. | |
| Source API | Yes : fully supported Support includes autotuning features (https://cloud.google.com/dataflow/service/dataflow-service-desc#autotuning-features). | Yes : fully supported | Yes : fully supported | Partially : bounded sources only Bounded sources run in the SDK harness (e.g. TextIO). Unbounded sources support is not implemented yet | Yes : fully supported | Partially : bounded source only Using Spark's DatasourceV2 API in microbatch mode (Continuous streaming mode is tagged experimental in spark and does not support aggregation). | Yes : fully supported | Yes : fully supported | Partially : bounded and unbounded, read by a single reader A source is split into exactly one part; reading several splits in parallel is not supported. | |
| Metrics | Partially Gauge metrics are not supported. All other metric types are supported. | Partially StringSet metrics are not supported. All other metric types are supported. | Partially : All metrics types are supported. Only attempted values are supported. No committed values for metrics. | No : not implemented Metrics are not collected or reported | Partially : All metric types are supported. Only attempted values are supported. No committed values for metrics. | Partially : All metric types are supported in batch mode. Only attempted values are supported. No committed values for metrics. | Partially : All metrics types supported, both in batching and streaming mode. Doesn't differentiate between committed and attempted values. | No : not implemented | Partially : attempted metrics only User metrics reported by the SDK harness surface as attempted values; committed values are not available. | |
| Stateful Processing | Partially : non-merging windows State is supported for non-merging windows. The MapState, SetState, and MultimapState state types are supported in the following scenarios: Java pipelines that don't use Streaming Engine; Java pipelines that use Streaming Engine and version 2.58.0 or later of the Java SDK. SetState, MapState, and MultimapState are not supported for pipelines that use Runner v2. | Yes : fully supported | Partially : non-merging windows State is supported for non-merging windows. SetState and MapState are not yet supported. | Partially : bag/value state and timers BagState and bag-backed ValueState are supported, together with event-time and processing-time timers. SetState, MapState, and ordered-list state are not yet supported. | Partially : full support in batch mode | Partially : full support in batch mode | Partially : non-merging windows | No : not implemented | No : not implemented User state is not wired to the harness. |
Last updated on 2026/10/12
Have you found everything you were looking for?
Was it all useful and clear? Is there anything that you would like to change? Let us know!