org.apache.beam.runners.spark.util

Class SparkCommon

java.lang.Object

org.apache.beam.runners.spark.util.SparkCommon

@Internal
public class SparkCommon
extends java.lang.Object

Common methods to build Spark specific objects used by different runners.

Constructor Summary

Constructors

Constructor and Description
SparkCommon()

Method Summary

Modifier and Type	Method and Description
static @Nullable org.apache.spark.scheduler.EventLoggingListener	startEventLoggingListener(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions pipelineOptions, long startTime) Starts an EventLoggingListener to save Beam Metrics on Spark's History Server if event logging is enabled.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkCommon

public SparkCommon()

Method Detail

startEventLoggingListener

public static @Nullable org.apache.spark.scheduler.EventLoggingListener startEventLoggingListener(org.apache.spark.api.java.JavaSparkContext jsc,
                                                                                                  SparkPipelineOptions pipelineOptions,
                                                                                                  long startTime)

Starts an EventLoggingListener to save Beam Metrics on Spark's History Server if event logging is enabled.

Returns:

The associated EventLoggingListener or null if it could not be started.