Package org.apache.beam.runners.spark.structuredstreaming.translation.helpers

Class EncoderHelpers

java.lang.Object
org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers
public class EncoderHelpers extends Object
Encoders utility class.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    EncoderHelpers.Utils
    Encoder / expression utils that are called from generated code.

  • Constructor Summary

    Constructors
    Constructor
    Description
    EncoderHelpers()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static <T> org.apache.spark.sql.Encoder<Collection<T>>
    collectionEncoder(org.apache.spark.sql.Encoder<T> enc)
    Creates a Spark Encoder of ArrayType for Java Collections with nullable elements.
    static <T> org.apache.spark.sql.Encoder<Collection<T>>
    collectionEncoder(org.apache.spark.sql.Encoder<T> enc, boolean nullable)
    Creates a Spark Encoder of ArrayType for Java Collections.
    static <T> org.apache.spark.sql.catalyst.expressions.Expression
    deserializeOneOf(org.apache.spark.sql.catalyst.expressions.Expression in, List<org.apache.spark.sql.Encoder<T>> encoders)
     
    static <T> org.apache.spark.sql.Encoder<T>
    encoderFor(Coder<T> coder)
    Creates a Spark Encoder for EncoderHelpers of BinaryType delegating to a Beam Coder underneath.
    static <T> org.apache.spark.sql.Encoder<T>
    encoderOf(Class<? super T> cls)
    Gets or creates a default Encoder for EncoderHelpers.
    static <K, V> org.apache.spark.sql.Encoder<KV<K,V>>
    kvEncoder(org.apache.spark.sql.Encoder<K> key, org.apache.spark.sql.Encoder<V> value)
    Creates a Spark Encoder for KV of StructType with fields key and value.
    static <MapT extends Map<K, V>, K, V>
    org.apache.spark.sql.Encoder<MapT>
    mapEncoder(org.apache.spark.sql.Encoder<K> key, org.apache.spark.sql.Encoder<V> value, Class<MapT> cls)
    Creates a Spark Encoder of MapType that deserializes to EncoderHelpers.
    static <T1, T2> org.apache.spark.sql.Encoder<org.apache.spark.util.MutablePair<T1,T2>>
    mutablePairEncoder(org.apache.spark.sql.Encoder<T1> enc1, org.apache.spark.sql.Encoder<T2> enc2)
    Creates a Spark Encoder for Spark's MutablePair of StructType with fields `_1` and `_2`.
    static <T> org.apache.spark.sql.Encoder<scala.Tuple2<Integer,T>>
    oneOfEncoder(List<org.apache.spark.sql.Encoder<T>> encoders)
    Creates a one-of Spark Encoder of StructType where each alternative is represented as colum / field named by its index with a separate Encoder each.
    static <T> org.apache.spark.sql.catalyst.expressions.Expression
    serializeOneOf(org.apache.spark.sql.catalyst.expressions.Expression in, List<org.apache.spark.sql.Encoder<T>> encoders)
     
    static <T, W extends BoundedWindow>
    org.apache.spark.sql.Encoder<WindowedValue<T>>
    windowedValueEncoder(org.apache.spark.sql.Encoder<T> value, org.apache.spark.sql.Encoder<W> window)
    Creates a Spark Encoder for EncoderHelpers of StructType with fields value, timestamp, windows and pane.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • EncoderHelpers

      public EncoderHelpers()

  • Method Details

    • encoderOf

      public static <T> org.apache.spark.sql.Encoder<T> encoderOf(Class<? super T> cls)
      Gets or creates a default Encoder for EncoderHelpers.

    • encoderFor

      public static <T> org.apache.spark.sql.Encoder<T> encoderFor(Coder<T> coder)
      Creates a Spark Encoder for EncoderHelpers of BinaryType delegating to a Beam Coder underneath.

      Note: For common types, if available, default Spark Encoders are used instead.

      Parameters:
      coder - Beam Coder

    • windowedValueEncoder

      public static <T, W extends BoundedWindow> org.apache.spark.sql.Encoder<WindowedValue<T>> windowedValueEncoder(org.apache.spark.sql.Encoder<T> value, org.apache.spark.sql.Encoder<W> window)
      Creates a Spark Encoder for EncoderHelpers of StructType with fields value, timestamp, windows and pane.
      Parameters:
      value - Encoder to encode field `value`.
      window - Encoder to encode individual windows in field `windows`

    • oneOfEncoder

      public static <T> org.apache.spark.sql.Encoder<scala.Tuple2<Integer,T>> oneOfEncoder(List<org.apache.spark.sql.Encoder<T>> encoders)
      Creates a one-of Spark Encoder of StructType where each alternative is represented as colum / field named by its index with a separate Encoder each.

      Externally this is represented as tuple (index, data) where an index corresponds to an Encoder in the provided list.

      Parameters:
      encoders - Encoders for each alternative.

    • kvEncoder

      public static <K, V> org.apache.spark.sql.Encoder<KV<K,V>> kvEncoder(org.apache.spark.sql.Encoder<K> key, org.apache.spark.sql.Encoder<V> value)
      Creates a Spark Encoder for KV of StructType with fields key and value.
      Parameters:
      key - Encoder to encode field `key`.
      value - Encoder to encode field `value`

    • collectionEncoder

      public static <T> org.apache.spark.sql.Encoder<Collection<T>> collectionEncoder(org.apache.spark.sql.Encoder<T> enc)
      Creates a Spark Encoder of ArrayType for Java Collections with nullable elements.
      Parameters:
      enc - Encoder to encode collection elements

    • collectionEncoder

      public static <T> org.apache.spark.sql.Encoder<Collection<T>> collectionEncoder(org.apache.spark.sql.Encoder<T> enc, boolean nullable)
      Creates a Spark Encoder of ArrayType for Java Collections.
      Parameters:
      enc - Encoder to encode collection elements
      nullable - Allow nullable collection elements

    • mapEncoder

      public static <MapT extends Map<K, V>, K, V> org.apache.spark.sql.Encoder<MapT> mapEncoder(org.apache.spark.sql.Encoder<K> key, org.apache.spark.sql.Encoder<V> value, Class<MapT> cls)
      Creates a Spark Encoder of MapType that deserializes to EncoderHelpers.
      Parameters:
      key - Encoder to encode keys
      value - Encoder to encode values
      cls - Specific class to use, supported are HashMap and TreeMap

    • mutablePairEncoder

      public static <T1, T2> org.apache.spark.sql.Encoder<org.apache.spark.util.MutablePair<T1,T2>> mutablePairEncoder(org.apache.spark.sql.Encoder<T1> enc1, org.apache.spark.sql.Encoder<T2> enc2)
      Creates a Spark Encoder for Spark's MutablePair of StructType with fields `_1` and `_2`.

      This is intended to be used in places such as aggregators.

      Parameters:
      enc1 - Encoder to encode `_1`
      enc2 - Encoder to encode `_2`

    • serializeOneOf

      public static <T> org.apache.spark.sql.catalyst.expressions.Expression serializeOneOf(org.apache.spark.sql.catalyst.expressions.Expression in, List<org.apache.spark.sql.Encoder<T>> encoders)

    • deserializeOneOf

      public static <T> org.apache.spark.sql.catalyst.expressions.Expression deserializeOneOf(org.apache.spark.sql.catalyst.expressions.Expression in, List<org.apache.spark.sql.Encoder<T>> encoders)