org.apache.beam.it.gcp.dataflow

Class AbstractPipelineLauncher

java.lang.Object

org.apache.beam.it.gcp.dataflow.AbstractPipelineLauncher

All Implemented Interfaces:

org.apache.beam.it.common.PipelineLauncher

Direct Known Subclasses:

ClassicTemplateClient, DefaultPipelineLauncher, FlexTemplateClient

public abstract class AbstractPipelineLauncher
extends java.lang.Object
implements org.apache.beam.it.common.PipelineLauncher

Abstract class covering the common operations between Classic and Flex templates.

Generally, the methods here are the ones that focus more on the Dataflow jobs rather than launching a specific type of template.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.it.common.PipelineLauncher

org.apache.beam.it.common.PipelineLauncher.JobState, org.apache.beam.it.common.PipelineLauncher.LaunchConfig, org.apache.beam.it.common.PipelineLauncher.LaunchInfo, org.apache.beam.it.common.PipelineLauncher.Sdk

Field Summary

Fields

Modifier and Type	Field and Description
protected Dataflow	client
protected java.util.List<java.lang.String>	launchedJobs
static java.lang.String	LEGACY_RUNNER
static java.lang.String	PARAM_JOB_ID
static java.lang.String	PARAM_JOB_TYPE
static java.lang.String	PARAM_RUNNER
static java.lang.String	RUNNER_V2

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractPipelineLauncher(Dataflow client)

Method Summary

Modifier and Type	Method and Description
Job	cancelJob(java.lang.String project, java.lang.String region, java.lang.String jobId)
void	cleanupAll()
Job	drainJob(java.lang.String project, java.lang.String region, java.lang.String jobId)
Job	getJob(java.lang.String project, java.lang.String region, java.lang.String jobId)
Job	getJob(java.lang.String project, java.lang.String region, java.lang.String jobId, java.lang.String view)
protected org.apache.beam.it.common.PipelineLauncher.LaunchInfo	getJobInfo(org.apache.beam.it.common.PipelineLauncher.LaunchConfig options, org.apache.beam.it.common.PipelineLauncher.JobState state, Job job) Creates a JobInfo object from the provided parameters.
protected org.apache.beam.it.common.PipelineLauncher.LaunchInfo.Builder	getJobInfoBuilder(org.apache.beam.it.common.PipelineLauncher.LaunchConfig options, org.apache.beam.it.common.PipelineLauncher.JobState state, Job job) Creates a JobInfo builder object from the provided parameters, enable derived class to add info incrementally.
org.apache.beam.it.common.PipelineLauncher.JobState	getJobStatus(java.lang.String project, java.lang.String region, java.lang.String jobId)
java.lang.Double	getMetric(java.lang.String project, java.lang.String region, java.lang.String jobId, java.lang.String metricName)
java.util.Map<java.lang.String,java.lang.Double>	getMetrics(java.lang.String project, java.lang.String region, java.lang.String jobId)
protected org.apache.beam.it.common.PipelineLauncher.JobState	handleJobState(Job job) Parses the job state if available or returns JobState#UNKNOWN if not given.
java.util.List<JobMessage>	listMessages(java.lang.String project, java.lang.String region, java.lang.String jobId, java.lang.String minimumImportance)
protected void	printJobResponse(Job job)
org.apache.beam.it.common.PipelineLauncher.JobState	waitUntilActive(java.lang.String project, java.lang.String region, java.lang.String jobId) Waits until the specified job is not in a pending state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.beam.it.common.PipelineLauncher

launch

Field Detail

LEGACY_RUNNER

public static final java.lang.String LEGACY_RUNNER

See Also:

Constant Field Values

RUNNER_V2

public static final java.lang.String RUNNER_V2

See Also:

Constant Field Values

PARAM_RUNNER

public static final java.lang.String PARAM_RUNNER

See Also:

Constant Field Values

PARAM_JOB_TYPE

public static final java.lang.String PARAM_JOB_TYPE

See Also:

Constant Field Values

PARAM_JOB_ID

public static final java.lang.String PARAM_JOB_ID

See Also:

Constant Field Values

launchedJobs

protected final java.util.List<java.lang.String> launchedJobs

client

protected final Dataflow client

Constructor Detail

AbstractPipelineLauncher

protected AbstractPipelineLauncher(Dataflow client)

Method Detail

getJob

public Job getJob(java.lang.String project,
                  java.lang.String region,
                  java.lang.String jobId)
           throws java.io.IOException

Specified by:

getJob in interface org.apache.beam.it.common.PipelineLauncher

Throws:

java.io.IOException

getJob

public Job getJob(java.lang.String project,
                  java.lang.String region,
                  java.lang.String jobId,
                  java.lang.String view)

Specified by:

getJob in interface org.apache.beam.it.common.PipelineLauncher

getJobStatus

public org.apache.beam.it.common.PipelineLauncher.JobState getJobStatus(java.lang.String project,
                                                                        java.lang.String region,
                                                                        java.lang.String jobId)
                                                                 throws java.io.IOException

Specified by:

getJobStatus in interface org.apache.beam.it.common.PipelineLauncher

Throws:

java.io.IOException

listMessages

public java.util.List<JobMessage> listMessages(java.lang.String project,
                                               java.lang.String region,
                                               java.lang.String jobId,
                                               java.lang.String minimumImportance)

Specified by:

listMessages in interface org.apache.beam.it.common.PipelineLauncher

cancelJob

public Job cancelJob(java.lang.String project,
                     java.lang.String region,
                     java.lang.String jobId)

Specified by:

cancelJob in interface org.apache.beam.it.common.PipelineLauncher

drainJob

public Job drainJob(java.lang.String project,
                    java.lang.String region,
                    java.lang.String jobId)

Specified by:

drainJob in interface org.apache.beam.it.common.PipelineLauncher

getMetric

@Nullable
public java.lang.Double getMetric(java.lang.String project,
                                            java.lang.String region,
                                            java.lang.String jobId,
                                            java.lang.String metricName)
                                     throws java.io.IOException

Specified by:

getMetric in interface org.apache.beam.it.common.PipelineLauncher

Throws:

java.io.IOException

getMetrics

public java.util.Map<java.lang.String,java.lang.Double> getMetrics(java.lang.String project,
                                                                   java.lang.String region,
                                                                   java.lang.String jobId)
                                                            throws java.io.IOException

Specified by:

getMetrics in interface org.apache.beam.it.common.PipelineLauncher

Throws:

java.io.IOException

printJobResponse

protected void printJobResponse(Job job)

handleJobState

protected org.apache.beam.it.common.PipelineLauncher.JobState handleJobState(Job job)

Parses the job state if available or returns JobState#UNKNOWN if not given.

getJobInfoBuilder

protected org.apache.beam.it.common.PipelineLauncher.LaunchInfo.Builder getJobInfoBuilder(org.apache.beam.it.common.PipelineLauncher.LaunchConfig options,
                                                                                          org.apache.beam.it.common.PipelineLauncher.JobState state,
                                                                                          Job job)

Creates a JobInfo builder object from the provided parameters, enable derived class to add info incrementally.

getJobInfo

protected final org.apache.beam.it.common.PipelineLauncher.LaunchInfo getJobInfo(org.apache.beam.it.common.PipelineLauncher.LaunchConfig options,
                                                                                 org.apache.beam.it.common.PipelineLauncher.JobState state,
                                                                                 Job job)

Creates a JobInfo object from the provided parameters.

waitUntilActive

public org.apache.beam.it.common.PipelineLauncher.JobState waitUntilActive(java.lang.String project,
                                                                           java.lang.String region,
                                                                           java.lang.String jobId)
                                                                    throws java.io.IOException

Waits until the specified job is not in a pending state.

Throws:

java.io.IOException

cleanupAll

public void cleanupAll()
                throws java.io.IOException

Specified by:

cleanupAll in interface org.apache.beam.it.common.PipelineLauncher

Throws:

java.io.IOException