org.apache.beam.sdk.testing

Class TestPipeline

java.lang.Object

org.apache.beam.sdk.Pipeline

org.apache.beam.sdk.testing.TestPipeline

All Implemented Interfaces:

org.junit.rules.TestRule

public class TestPipeline
extends Pipeline
implements org.junit.rules.TestRule

A creator of test pipelines that can be used inside of tests that can be configured to run locally or against a remote pipeline runner.

It is recommended to tag hand-selected tests for this purpose using the ValidatesRunner Category annotation, as each test run against a pipeline runner will utilize resources of that pipeline runner.

In order to run tests on a pipeline runner, the following conditions must be met:

• System property "beamTestPipelineOptions" must contain a JSON delimited list of pipeline options. For example:

   [
       "--runner=TestDataflowRunner",
       "--project=mygcpproject",
       "--stagingLocation=gs://mygcsbucket/path"
       ]

  Note that the set of pipeline options required is pipeline runner specific.
• Jars containing the SDK and test classes must be available on the classpath.

Use PAssert for tests, as it integrates with this test harness in both direct and remote execution modes. For example:

  @Rule
 public final transient TestPipeline p = TestPipeline.create();

  @Test
  @Category(NeedsRunner.class)
 public void myPipelineTest() throws Exception {
   final PCollection<String> pCollection = pipeline.apply(...)
   PAssert.that(pCollection).containsInAnyOrder(...);
   pipeline.run();
 }
 

For pipeline runners, it is required that they must throw an AssertionError containing the message from the PAssert that failed.

See also the Testing documentation section.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TestPipeline.AbandonedNodeException An exception thrown in case an abandoned PTransform is detected, that is, a PTransform that has not been run.
static class	TestPipeline.PipelineRunMissingException An exception thrown in case a test finishes without invoking Pipeline.run().

Nested classes/interfaces inherited from class org.apache.beam.sdk.Pipeline

Pipeline.PipelineExecutionException, Pipeline.PipelineVisitor

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	PROPERTY_BEAM_TEST_PIPELINE_OPTIONS System property used to set TestPipelineOptions.

Method Summary

Modifier and Type	Method and Description
org.junit.runners.model.Statement	apply(org.junit.runners.model.Statement statement, Description description)
static java.lang.String[]	convertToArgs(PipelineOptions options)
static TestPipeline	create() Creates and returns a new test pipeline.
TestPipeline	enableAbandonedNodeEnforcement(boolean enable) Enables the abandoned node detection.
TestPipeline	enableAutoRunIfMissing(boolean enable) If enabled, a pipeline.run() statement will be added automatically in case it is missing in the test.
static TestPipeline	fromOptions(PipelineOptions options)
PipelineOptions	getOptions()
PipelineResult	run() Runs this TestPipeline, unwrapping any AssertionError that is raised during testing.
static PipelineOptions	testingPipelineOptions() Creates PipelineOptions for testing.
java.lang.String	toString()
static void	verifyPAssertsSucceeded(Pipeline pipeline, PipelineResult pipelineResult) Verifies all {PAsserts} in the pipeline have been executed and were successful.

Methods inherited from class org.apache.beam.sdk.Pipeline

apply, apply, applyTransform, applyTransform, begin, create, getCoderRegistry, replaceAll, run, setCoderRegistry, traverseTopologically

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

PROPERTY_BEAM_TEST_PIPELINE_OPTIONS

public static final java.lang.String PROPERTY_BEAM_TEST_PIPELINE_OPTIONS

System property used to set TestPipelineOptions.

See Also:

Constant Field Values

Method Detail

create

public static TestPipeline create()

Creates and returns a new test pipeline.

Use PAssert to add tests, then call Pipeline.run() to execute the pipeline and check the tests.

fromOptions

public static TestPipeline fromOptions(PipelineOptions options)

getOptions

public PipelineOptions getOptions()

apply

public org.junit.runners.model.Statement apply(org.junit.runners.model.Statement statement,
                                               Description description)

Specified by:

apply in interface org.junit.rules.TestRule

run

public PipelineResult run()

Runs this TestPipeline, unwrapping any AssertionError that is raised during testing.

Overrides:

run in class Pipeline

enableAbandonedNodeEnforcement

public TestPipeline enableAbandonedNodeEnforcement(boolean enable)

Enables the abandoned node detection. Abandoned nodes are PTransforms, PAsserts included, that were not executed by the pipeline runner. Abandoned nodes are most likely to occur due to the one of the following scenarios:

• Lack of a pipeline.run() statement at the end of a test.
• Addition of PTransforms after the pipeline has already run.

Abandoned node detection is automatically enabled when a real pipeline runner (i.e. not a CrashingRunner) and/or a NeedsRunner or a ValidatesRunner annotation are detected.

enableAutoRunIfMissing

public TestPipeline enableAutoRunIfMissing(boolean enable)

If enabled, a pipeline.run() statement will be added automatically in case it is missing in the test.

toString

public java.lang.String toString()

Overrides:

toString in class Pipeline

testingPipelineOptions

public static PipelineOptions testingPipelineOptions()

Creates PipelineOptions for testing.

convertToArgs

public static java.lang.String[] convertToArgs(PipelineOptions options)

verifyPAssertsSucceeded

public static void verifyPAssertsSucceeded(Pipeline pipeline,
                                           PipelineResult pipelineResult)

Verifies all {PAsserts} in the pipeline have been executed and were successful.

Note this only runs for runners which support Metrics. Runners which do not should verify this in some other way. See: https://issues.apache.org/jira/browse/BEAM-2001