org.apache.beam.runners.spark.util

Class SparkCompat

java.lang.Object

org.apache.beam.runners.spark.util.SparkCompat

public class SparkCompat
extends java.lang.Object

A set of functions to provide API compatibility between Spark 2 and Spark 3.

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.scheduler.SparkListenerApplicationStart	buildSparkListenerApplicationStart(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, long time, PipelineResult result)
static <K,InputT,AccumT,OutputT> org.apache.spark.api.java.JavaPairRDD<K,org.apache.beam.sdk.util.WindowedValue<OutputT>>	extractOutput(org.apache.spark.api.java.JavaPairRDD<K,<any>> accumulatePerKey, <any> sparkCombineFn) Extracts the output for a given collection of WindowedAccumulators.
static <T> org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<T>>	joinStreams(org.apache.spark.streaming.api.java.JavaStreamingContext streamingContext, java.util.List<org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<T>>> dStreams) Union of dStreams in the given StreamingContext.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

joinStreams

public static <T> org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<T>> joinStreams(org.apache.spark.streaming.api.java.JavaStreamingContext streamingContext,
                                                                                                                         java.util.List<org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<T>>> dStreams)

Union of dStreams in the given StreamingContext.

This is required because the API to join (union) DStreams is different among Spark versions. See https://issues.apache.org/jira/browse/SPARK-25737

extractOutput

public static <K,InputT,AccumT,OutputT> org.apache.spark.api.java.JavaPairRDD<K,org.apache.beam.sdk.util.WindowedValue<OutputT>> extractOutput(org.apache.spark.api.java.JavaPairRDD<K,<any>> accumulatePerKey,
                                                                                                                                               <any> sparkCombineFn)

Extracts the output for a given collection of WindowedAccumulators.

This is required because the API of JavaPairRDD.flatMapValues is different among Spark versions. See https://issues.apache.org/jira/browse/SPARK-19287

buildSparkListenerApplicationStart

public static org.apache.spark.scheduler.SparkListenerApplicationStart buildSparkListenerApplicationStart(org.apache.spark.api.java.JavaSparkContext jsc,
                                                                                                          SparkPipelineOptions options,
                                                                                                          long time,
                                                                                                          PipelineResult result)