org.apache.beam.runners.spark.util

Class SparkCompat

java.lang.Object

org.apache.beam.runners.spark.util.SparkCompat

public class SparkCompat
extends java.lang.Object

A set of functions to provide API compatibility between Spark 2 and Spark 3.

Constructor Summary

Constructors

Constructor and Description
SparkCompat()

Method Summary

Modifier and Type	Method and Description
static <K,InputT,AccumT,OutputT> org.apache.spark.api.java.JavaPairRDD<K,org.apache.beam.sdk.util.WindowedValue<OutputT>>	extractOutput(org.apache.spark.api.java.JavaPairRDD<K,<any>> accumulatePerKey, <any> sparkCombineFn) Extracts the output for a given collection of WindowedAccumulators.
static <T> org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<T>>	joinStreams(org.apache.spark.streaming.api.java.JavaStreamingContext streamingContext, java.util.List<org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<T>>> dStreams) Union of dStreams in the given StreamingContext.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkCompat

public SparkCompat()

Method Detail

joinStreams

public static <T> org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<T>> joinStreams(org.apache.spark.streaming.api.java.JavaStreamingContext streamingContext,
                                                                                                                         java.util.List<org.apache.spark.streaming.api.java.JavaDStream<org.apache.beam.sdk.util.WindowedValue<T>>> dStreams)

Union of dStreams in the given StreamingContext.

This is required because the API to join (union) DStreams is different among Spark versions. See https://issues.apache.org/jira/browse/SPARK-25737

extractOutput

public static <K,InputT,AccumT,OutputT> org.apache.spark.api.java.JavaPairRDD<K,org.apache.beam.sdk.util.WindowedValue<OutputT>> extractOutput(org.apache.spark.api.java.JavaPairRDD<K,<any>> accumulatePerKey,
                                                                                                                                               <any> sparkCombineFn)

Extracts the output for a given collection of WindowedAccumulators.

This is required because the API of JavaPairRDD.flatMapValues is different among Spark versions. See https://issues.apache.org/jira/browse/SPARK-19287