See: Description
| Interface | Description |
|---|---|
| SparkCommonPipelineOptions |
Spark runner
PipelineOptions handles Spark execution-related configurations, such as the
master address, and other user-related knobs. |
| SparkContextOptions |
A custom
PipelineOptions to work with properties related to JavaSparkContext. |
| SparkPipelineOptions |
Spark runner
PipelineOptions handles Spark execution-related configurations, such as the
master address, batch-interval, and other user-related knobs. |
| SparkPortableStreamingPipelineOptions |
Pipeline options specific to the Spark portable runner running a streaming job.
|
| TestSparkPipelineOptions |
A
SparkPipelineOptions for tests. |
| Class | Description |
|---|---|
| SparkCommonPipelineOptions.StorageLevelFactory |
Returns Spark's default storage level for the Dataset or RDD API based on the respective
runner.
|
| SparkCommonPipelineOptions.TmpCheckpointDirFactory |
Returns the default checkpoint directory of /tmp/${job.name}.
|
| SparkContextOptions.EmptyListenersList |
Returns an empty list, to avoid handling null.
|
| SparkJobInvoker |
Creates a job invocation to manage the Spark runner's execution of a portable pipeline.
|
| SparkJobServerDriver |
Driver program that starts a job server for the Spark runner.
|
| SparkJobServerDriver.SparkServerConfiguration |
Spark runner-specific Configuration for the jobServer.
|
| SparkNativePipelineVisitor |
Pipeline visitor for translating a Beam pipeline into equivalent Spark operations.
|
| SparkPipelineResult |
Represents a Spark pipeline execution result.
|
| SparkPipelineRunner |
Runs a portable pipeline on Apache Spark.
|
| SparkRunner |
The SparkRunner translate operations defined on a pipeline to a representation executable by
Spark, and then submitting the job to Spark to be executed.
|
| SparkRunner.Evaluator |
Evaluator on the pipeline.
|
| SparkRunnerDebugger |
Pipeline runner which translates a Beam pipeline into equivalent Spark operations, without
running them.
|
| SparkRunnerDebugger.DebugSparkPipelineResult |
PipelineResult of running a
Pipeline using SparkRunnerDebugger Use SparkRunnerDebugger.DebugSparkPipelineResult.getDebugString() to get a String representation of the Pipeline translated into
Spark native operations. |
| SparkRunnerRegistrar | |
| SparkRunnerRegistrar.Options |
Registers the
SparkPipelineOptions. |
| SparkRunnerRegistrar.Runner |
Registers the
SparkRunner. |
| TestSparkPipelineOptions.DefaultStopPipelineWatermarkFactory |
A factory to provide the default watermark to stop a pipeline that reads from an unbounded
source.
|
| TestSparkRunner |
The SparkRunner translate operations defined on a pipeline to a representation executable by
Spark, and then submitting the job to Spark to be executed.
|