org.apache.beam.runners.spark.structuredstreaming.translation.helpers

Class EncoderHelpers

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers

public class EncoderHelpers
extends java.lang.Object

Encoders utility class.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	EncoderHelpers.Utils Encoder / expression utils that are called from generated code.

Constructor Summary

Constructors

Constructor and Description
EncoderHelpers()

Method Summary

Modifier and Type	Method and Description
static <T> org.apache.spark.sql.Encoder<java.util.Collection<T>>	collectionEncoder(org.apache.spark.sql.Encoder<T> enc) Creates a Spark Encoder of ArrayType for Java Collections with nullable elements.
static <T> org.apache.spark.sql.Encoder<java.util.Collection<T>>	collectionEncoder(org.apache.spark.sql.Encoder<T> enc, boolean nullable) Creates a Spark Encoder of ArrayType for Java Collections.
static <T> org.apache.spark.sql.catalyst.expressions.Expression	deserializeOneOf(org.apache.spark.sql.catalyst.expressions.Expression in, java.util.List<org.apache.spark.sql.Encoder<T>> encoders)
static <T> org.apache.spark.sql.Encoder<T>	encoderFor(Coder<T> coder) Creates a Spark Encoder for T of BinaryType delegating to a Beam Coder underneath.
static <T> org.apache.spark.sql.Encoder<T>	encoderOf(java.lang.Class<? super T> cls) Gets or creates a default Encoder for T.
static <K,V> org.apache.spark.sql.Encoder<KV<K,V>>	kvEncoder(org.apache.spark.sql.Encoder<K> key, org.apache.spark.sql.Encoder<V> value) Creates a Spark Encoder for KV of StructType with fields key and value.
static <MapT extends java.util.Map<K,V>,K,V> org.apache.spark.sql.Encoder<MapT>	mapEncoder(org.apache.spark.sql.Encoder<K> key, org.apache.spark.sql.Encoder<V> value, java.lang.Class<MapT> cls) Creates a Spark Encoder of MapType that deserializes to MapT.
static <T1,T2> org.apache.spark.sql.Encoder<org.apache.spark.util.MutablePair<T1,T2>>	mutablePairEncoder(org.apache.spark.sql.Encoder<T1> enc1, org.apache.spark.sql.Encoder<T2> enc2) Creates a Spark Encoder for Spark's MutablePair of StructType with fields `_1` and `_2`.
static <T> org.apache.spark.sql.Encoder<scala.Tuple2<java.lang.Integer,T>>	oneOfEncoder(java.util.List<org.apache.spark.sql.Encoder<T>> encoders) Creates a one-of Spark Encoder of StructType where each alternative is represented as colum / field named by its index with a separate Encoder each.
static <T> org.apache.spark.sql.catalyst.expressions.Expression	serializeOneOf(org.apache.spark.sql.catalyst.expressions.Expression in, java.util.List<org.apache.spark.sql.Encoder<T>> encoders)
static <T,W extends BoundedWindow> org.apache.spark.sql.Encoder<org.apache.beam.sdk.util.WindowedValue<T>>	windowedValueEncoder(org.apache.spark.sql.Encoder<T> value, org.apache.spark.sql.Encoder<W> window) Creates a Spark Encoder for T of StructType with fields value, timestamp, windows and pane.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EncoderHelpers

public EncoderHelpers()

Method Detail

encoderOf

public static <T> org.apache.spark.sql.Encoder<T> encoderOf(java.lang.Class<? super T> cls)

Gets or creates a default Encoder for T.

encoderFor

public static <T> org.apache.spark.sql.Encoder<T> encoderFor(Coder<T> coder)

Creates a Spark Encoder for T of BinaryType delegating to a Beam Coder underneath.

Note: For common types, if available, default Spark Encoders are used instead.

Parameters:

coder - Beam Coder

windowedValueEncoder

public static <T,W extends BoundedWindow> org.apache.spark.sql.Encoder<org.apache.beam.sdk.util.WindowedValue<T>> windowedValueEncoder(org.apache.spark.sql.Encoder<T> value,
                                                                                                                                       org.apache.spark.sql.Encoder<W> window)

Creates a Spark Encoder for T of StructType with fields value, timestamp, windows and pane.

Parameters:

value - Encoder to encode field `value`.

window - Encoder to encode individual windows in field `windows`

oneOfEncoder

public static <T> org.apache.spark.sql.Encoder<scala.Tuple2<java.lang.Integer,T>> oneOfEncoder(java.util.List<org.apache.spark.sql.Encoder<T>> encoders)

Creates a one-of Spark Encoder of StructType where each alternative is represented as colum / field named by its index with a separate Encoder each.

Externally this is represented as tuple (index, data) where an index corresponds to an Encoder in the provided list.

Parameters:

encoders - Encoders for each alternative.

kvEncoder

public static <K,V> org.apache.spark.sql.Encoder<KV<K,V>> kvEncoder(org.apache.spark.sql.Encoder<K> key,
                                                                    org.apache.spark.sql.Encoder<V> value)

Creates a Spark Encoder for KV of StructType with fields key and value.

Parameters:

key - Encoder to encode field `key`.

value - Encoder to encode field `value`

collectionEncoder

public static <T> org.apache.spark.sql.Encoder<java.util.Collection<T>> collectionEncoder(org.apache.spark.sql.Encoder<T> enc)

Creates a Spark Encoder of ArrayType for Java Collections with nullable elements.

Parameters:

enc - Encoder to encode collection elements

collectionEncoder

public static <T> org.apache.spark.sql.Encoder<java.util.Collection<T>> collectionEncoder(org.apache.spark.sql.Encoder<T> enc,
                                                                                          boolean nullable)

Creates a Spark Encoder of ArrayType for Java Collections.

Parameters:

enc - Encoder to encode collection elements

nullable - Allow nullable collection elements

mapEncoder

public static <MapT extends java.util.Map<K,V>,K,V> org.apache.spark.sql.Encoder<MapT> mapEncoder(org.apache.spark.sql.Encoder<K> key,
                                                                                                  org.apache.spark.sql.Encoder<V> value,
                                                                                                  java.lang.Class<MapT> cls)

Creates a Spark Encoder of MapType that deserializes to MapT.

Parameters:

key - Encoder to encode keys

value - Encoder to encode values

cls - Specific class to use, supported are HashMap and TreeMap

mutablePairEncoder

public static <T1,T2> org.apache.spark.sql.Encoder<org.apache.spark.util.MutablePair<T1,T2>> mutablePairEncoder(org.apache.spark.sql.Encoder<T1> enc1,
                                                                                                                org.apache.spark.sql.Encoder<T2> enc2)

Creates a Spark Encoder for Spark's MutablePair of StructType with fields `_1` and `_2`.

This is intended to be used in places such as aggregators.

Parameters:

enc1 - Encoder to encode `_1`

enc2 - Encoder to encode `_2`

serializeOneOf

public static <T> org.apache.spark.sql.catalyst.expressions.Expression serializeOneOf(org.apache.spark.sql.catalyst.expressions.Expression in,
                                                                                      java.util.List<org.apache.spark.sql.Encoder<T>> encoders)

deserializeOneOf

public static <T> org.apache.spark.sql.catalyst.expressions.Expression deserializeOneOf(org.apache.spark.sql.catalyst.expressions.Expression in,
                                                                                        java.util.List<org.apache.spark.sql.Encoder<T>> encoders)