back to collapsed details

Additional common features not yet part of the Beam model

Drain
Checkpoint
Google Cloud DataflowApache FlinkApache Spark (RDD/DStream based)Apache Spark Structured Streaming (Dataset based)IBM StreamsApache SamzaApache NemoHazelcast JetTwister2Python Direct FnRunnerGo Direct Runner

Partially :


Dataflow has a native drain operation, but it does not work in the presence of event time timer loops. Final implemention pending model support.

Partially :


Flink supports taking a "savepoint" of the pipeline and shutting the pipeline down after its completion.

:


:


:


:


:


:


:


No :


Partially :


Flink has a native savepoint capability.

Partially :


Spark has a native savepoint capability.

No :


not implemented

:


Partially :


Samza has a native checkpoint capability.

:


:


:


Last updated on 2021/02/05

Have you found everything you were looking for?

Was it all useful and clear? Is there anything that you would like to change? Let us know!