org.apache.beam.runners.spark

Class SparkRunnerDebugger

java.lang.Object

org.apache.beam.sdk.PipelineRunner<SparkPipelineResult>

org.apache.beam.runners.spark.SparkRunnerDebugger

public final class SparkRunnerDebugger
extends PipelineRunner<SparkPipelineResult>

Pipeline runner which translates a Beam pipeline into equivalent Spark operations, without running them. Used for debugging purposes.

Example:

 SparkPipelineOptions options = PipelineOptionsFactory.as(SparkPipelineOptions.class);
 options.setRunner(SparkRunnerDebugger.class);
 Pipeline pipeline = Pipeline.create(options);
 SparkRunnerDebugger.DebugSparkPipelineResult result =
     (SparkRunnerDebugger.DebugSparkPipelineResult) pipeline.run();
 String sparkPipeline = result.getDebugString();
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SparkRunnerDebugger.DebugSparkPipelineResult PipelineResult of running a Pipeline using SparkRunnerDebugger Use SparkRunnerDebugger.DebugSparkPipelineResult.getDebugString() to get a String representation of the Pipeline translated into Spark native operations.

Method Summary

Modifier and Type	Method and Description
static SparkRunnerDebugger	fromOptions(PipelineOptions options)
SparkPipelineResult	run(Pipeline pipeline) Processes the given Pipeline, potentially asynchronously, returning a runner-specific type of result.

Methods inherited from class org.apache.beam.sdk.PipelineRunner

create, run, run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

fromOptions

public static SparkRunnerDebugger fromOptions(PipelineOptions options)

run

public SparkPipelineResult run(Pipeline pipeline)

Description copied from class: PipelineRunner

Processes the given Pipeline, potentially asynchronously, returning a runner-specific type of result.

Specified by:

run in class PipelineRunner<SparkPipelineResult>