Package org.apache.beam.runners.spark.structuredstreaming.io

Class BoundedDatasetFactory

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.io.BoundedDatasetFactory

public class BoundedDatasetFactory
extends Object

Method Summary

Modifier and Type	Method	Description
static <T> org.apache.spark.sql.Dataset<WindowedValue<T>>	createDatasetFromRDD(org.apache.spark.sql.SparkSession session, BoundedSource<T> source, Supplier<PipelineOptions> options, org.apache.spark.sql.Encoder<WindowedValue<T>> encoder)	Create a Dataset for a BoundedSource via a Spark RDD.
static <T> org.apache.spark.sql.Dataset<WindowedValue<T>>	createDatasetFromRows(org.apache.spark.sql.SparkSession session, BoundedSource<T> source, Supplier<PipelineOptions> options, org.apache.spark.sql.Encoder<WindowedValue<T>> encoder)	Create a Dataset for a BoundedSource via a Spark Table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

createDatasetFromRows

public static <T>
org.apache.spark.sql.Dataset<WindowedValue<T>> createDatasetFromRows(org.apache.spark.sql.SparkSession session,
 BoundedSource<T> source,
 Supplier<PipelineOptions> options,
 org.apache.spark.sql.Encoder<WindowedValue<T>> encoder)

Create a Dataset for a BoundedSource via a Spark Table.

Unfortunately tables are expected to return an InternalRow, requiring serialization. This makes this approach at the time being significantly less performant than creating a dataset from an RDD.

createDatasetFromRDD

public static <T>
org.apache.spark.sql.Dataset<WindowedValue<T>> createDatasetFromRDD(org.apache.spark.sql.SparkSession session,
 BoundedSource<T> source,
 Supplier<PipelineOptions> options,
 org.apache.spark.sql.Encoder<WindowedValue<T>> encoder)

Create a Dataset for a BoundedSource via a Spark RDD.

This is currently the most efficient approach as it avoid any serialization overhead.