org.apache.beam.runners.dataflow

Class DataflowPipelineJob

java.lang.Object

org.apache.beam.runners.dataflow.DataflowPipelineJob

All Implemented Interfaces:

PipelineResult

Direct Known Subclasses:

DataflowTemplateJob

public class DataflowPipelineJob
extends java.lang.Object
implements PipelineResult

A DataflowPipelineJob represents a job submitted to Dataflow using DataflowRunner.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.PipelineResult

PipelineResult.State

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String	jobId The id for the job.
protected static org.apache.beam.sdk.util.FluentBackoff	STATUS_BACKOFF_FACTORY
protected BiMap<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String>	transformStepNames

Constructor Summary

Constructors

Constructor and Description
DataflowPipelineJob(DataflowClient dataflowClient, java.lang.String jobId, DataflowPipelineOptions dataflowOptions, java.util.Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames) Constructs the job.

Method Summary

Modifier and Type	Method and Description
PipelineResult.State	cancel() Cancels the pipeline execution.
DataflowPipelineOptions	getDataflowOptions()
java.lang.String	getJobId() Get the id of this job.
java.lang.String	getProjectId() Get the project this job exists in.
java.lang.String	getRegion() Get the region this job exists in.
DataflowPipelineJob	getReplacedByJob() Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.
PipelineResult.State	getState() Retrieves the current state of the pipeline execution.
MetricResults	metrics() Returns the object to access metrics from the pipeline.
PipelineResult.State	waitUntilFinish() Waits until the pipeline finishes and returns the final status.
PipelineResult.State	waitUntilFinish(Duration duration) Waits until the pipeline finishes and returns the final status.
PipelineResult.State	waitUntilFinish(Duration duration, MonitoringUtil.JobMessagesHandler messageHandler) Waits until the pipeline finishes and returns the final status.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

jobId

protected java.lang.String jobId

The id for the job.

transformStepNames

protected BiMap<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames

STATUS_BACKOFF_FACTORY

protected static final org.apache.beam.sdk.util.FluentBackoff STATUS_BACKOFF_FACTORY

Constructor Detail

DataflowPipelineJob

public DataflowPipelineJob(DataflowClient dataflowClient,
                           java.lang.String jobId,
                           DataflowPipelineOptions dataflowOptions,
                           java.util.Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames)

Constructs the job.

Parameters:

jobId - the job id

dataflowOptions - used to configure the client for the Dataflow Service

transformStepNames - a mapping from AppliedPTransforms to Step Names

Method Detail

getJobId

public java.lang.String getJobId()

Get the id of this job.

getProjectId

public java.lang.String getProjectId()

Get the project this job exists in.

getDataflowOptions

public DataflowPipelineOptions getDataflowOptions()

getRegion

public java.lang.String getRegion()

Get the region this job exists in.

getReplacedByJob

public DataflowPipelineJob getReplacedByJob()

Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.

Throws:

java.lang.IllegalStateException - if called before the job has terminated or if the job terminated but was not updated

waitUntilFinish

@Nullable
public PipelineResult.State waitUntilFinish()

Description copied from interface: PipelineResult

Waits until the pipeline finishes and returns the final status.

Specified by:

waitUntilFinish in interface PipelineResult

Returns:

The final state of the pipeline.

waitUntilFinish

@Nullable
public PipelineResult.State waitUntilFinish(Duration duration)

Description copied from interface: PipelineResult

Waits until the pipeline finishes and returns the final status. It times out after the given duration.

Specified by:

waitUntilFinish in interface PipelineResult

Parameters:

duration - The time to wait for the pipeline to finish. Provide a value less than 1 ms for an infinite wait.

Returns:

The final state of the pipeline or null on timeout.

waitUntilFinish

@Nullable
public PipelineResult.State waitUntilFinish(Duration duration,
                                                      MonitoringUtil.JobMessagesHandler messageHandler)
                                               throws java.io.IOException,
                                                      java.lang.InterruptedException

Waits until the pipeline finishes and returns the final status.

Parameters:

duration - The time to wait for the job to finish. Provide a value less than 1 ms for an infinite wait.

messageHandler - If non null this handler will be invoked for each batch of messages received.

Returns:

The final state of the job or null on timeout or if the thread is interrupted.

Throws:

java.io.IOException - If there is a persistent problem getting job information.

java.lang.InterruptedException

cancel

public PipelineResult.State cancel()
                            throws java.io.IOException

Description copied from interface: PipelineResult

Cancels the pipeline execution.

Specified by:

cancel in interface PipelineResult

Throws:

java.io.IOException - if there is a problem executing the cancel request.

getState

public PipelineResult.State getState()

Description copied from interface: PipelineResult

Retrieves the current state of the pipeline execution.

Specified by:

getState in interface PipelineResult

Returns:

the PipelineResult.State representing the state of this pipeline.

metrics

public MetricResults metrics()

Description copied from interface: PipelineResult

Returns the object to access metrics from the pipeline.

Specified by:

metrics in interface PipelineResult