Package org.apache.beam.runners.spark.structuredstreaming.translation

package org.apache.beam.runners.spark.structuredstreaming.translation

Internal translators for running Beam pipelines on Spark.

Related Packages

Package	Description
org.apache.beam.runners.spark.structuredstreaming	Internal implementation of the Beam runner for Apache Spark.
org.apache.beam.runners.spark.structuredstreaming.translation.batch	Internal utilities to translate Beam pipelines to Spark batching.
org.apache.beam.runners.spark.structuredstreaming.translation.helpers	Internal helpers to translate Beam pipelines to Spark streaming.
org.apache.beam.runners.spark.structuredstreaming.translation.streaming
org.apache.beam.runners.spark.structuredstreaming.translation.utils	Internal utils to translate Beam pipelines to Spark streaming.
org.apache.beam.runners.spark.structuredstreaming.examples
org.apache.beam.runners.spark.structuredstreaming.io	Spark-specific transforms for I/O.
org.apache.beam.runners.spark.structuredstreaming.metrics	Provides internal utilities for implementing Beam metrics using Spark accumulators.

Class	Description
EvaluationContext	The EvaluationContext is the result of a pipeline translation and can be used to evaluate / run the pipeline.
EvaluationContext.NamedDataset<T>
PipelineTranslator	The pipeline translator translates a Beam Pipeline into a Spark correspondence, that can then be evaluated.
PipelineTranslator.TranslationState	Shared, mutable state during the translation of a pipeline and omitted afterwards.
PipelineTranslator.UnresolvedTranslation<InT,T>	Unresolved translation, allowing to optimize the generated Spark DAG.
PipelineTranslatorFactory	Factory to create the PipelineTranslator matching the execution mode of the pipeline.
PipelineTranslatorStreaming	Pipeline translator for streaming pipelines on Spark 4.
SparkSessionFactory
SparkSessionFactory.SparkKryoRegistrator	KryoRegistrator for Spark to serialize broadcast variables used for side-inputs.
StreamingEvaluationContext	Starts one Spark Structured Streaming query per leaf dataset and blocks until all of them reach a terminal state.
TransformTranslator<InT extends PInput,OutT extends POutput,TransformT extends PTransform<InT,OutT>>	A TransformTranslator provides the capability to translate a specific primitive or composite PTransform into its Spark correspondence.