Package org.apache.beam.runners.spark.structuredstreaming.translation

package org.apache.beam.runners.spark.structuredstreaming.translation

Internal translators for running Beam pipelines on Spark.

Related Packages

Package	Description
org.apache.beam.runners.spark.structuredstreaming	Internal implementation of the Beam runner for Apache Spark.
org.apache.beam.runners.spark.structuredstreaming.translation.batch	Internal utilities to translate Beam pipelines to Spark batching.
org.apache.beam.runners.spark.structuredstreaming.translation.helpers	Internal helpers to translate Beam pipelines to Spark streaming.
org.apache.beam.runners.spark.structuredstreaming.translation.utils	Internal utils to translate Beam pipelines to Spark streaming.
org.apache.beam.runners.spark.structuredstreaming.examples
org.apache.beam.runners.spark.structuredstreaming.io	Spark-specific transforms for I/O.
org.apache.beam.runners.spark.structuredstreaming.metrics	Provides internal utilities for implementing Beam metrics using Spark accumulators.

Class	Description
EvaluationContext	The EvaluationContext is the result of a pipeline translation and can be used to evaluate / run the pipeline.
PipelineTranslator	The pipeline translator translates a Beam Pipeline into a Spark correspondence, that can then be evaluated.
PipelineTranslator.TranslationState	Shared, mutable state during the translation of a pipeline and omitted afterwards.
PipelineTranslator.UnresolvedTranslation<InT,T>	Unresolved translation, allowing to optimize the generated Spark DAG.
SparkSessionFactory
SparkSessionFactory.SparkKryoRegistrator	KryoRegistrator for Spark to serialize broadcast variables used for side-inputs.
TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<InT,OutT>>	A TransformTranslator provides the capability to translate a specific primitive or composite PTransform into its Spark correspondence.