Package org.apache.beam.runners.spark.structuredstreaming.translation

Class SparkSessionFactory

java.lang.Object

org.apache.beam.runners.spark.structuredstreaming.translation.SparkSessionFactory

public class SparkSessionFactory
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	SparkSessionFactory.SparkKryoRegistrator	KryoRegistrator for Spark to serialize broadcast variables used for side-inputs.

Constructor Summary

Constructors

Constructor	Description
SparkSessionFactory()

Method Summary

Modifier and Type	Method	Description
static org.apache.spark.sql.SparkSession	acquire(SparkStructuredStreamingPipelineOptions options)	Returns the SparkSession for a pipeline, paired with release(org.apache.spark.sql.SparkSession).
static void	release(org.apache.spark.sql.SparkSession session)	Releases a session from acquire(org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions) and stops it when no longer used.
static org.apache.spark.sql.SparkSession.Builder	sessionBuilder(String master)	Creates Spark session builder with some optimizations for local mode, e.g.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SparkSessionFactory

public SparkSessionFactory()

Method Details

acquire

public static org.apache.spark.sql.SparkSession acquire(SparkStructuredStreamingPipelineOptions options)

Returns the SparkSession for a pipeline, paired with release(org.apache.spark.sql.SparkSession).

release

public static void release(org.apache.spark.sql.SparkSession session)

Releases a session from acquire(org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions) and stops it when no longer used. The stop runs under the lock, a pipeline starting meanwhile creates a new session.

sessionBuilder

public static org.apache.spark.sql.SparkSession.Builder sessionBuilder(String master)

Creates Spark session builder with some optimizations for local mode, e.g. in tests.