org.apache.beam.runners.dataflow

Class DataflowPipelineJob

java.lang.Object

org.apache.beam.runners.dataflow.DataflowPipelineJob

All Implemented Interfaces:

PipelineResult

Direct Known Subclasses:

DataflowTemplateJob

public class DataflowPipelineJob
extends java.lang.Object
implements PipelineResult

A DataflowPipelineJob represents a job submitted to Dataflow using DataflowRunner.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.PipelineResult

PipelineResult.State

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.apache.beam.sdk.util.FluentBackoff	STATUS_BACKOFF_FACTORY

Constructor Summary

Constructors

Constructor and Description
DataflowPipelineJob(DataflowClient dataflowClient, java.lang.String jobId, DataflowPipelineOptions dataflowOptions, java.util.Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames) Constructs the job.
DataflowPipelineJob(DataflowClient dataflowClient, java.lang.String jobId, DataflowPipelineOptions dataflowOptions, java.util.Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames, @Nullable org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipelineProto) Constructs the job.

Method Summary

Modifier and Type	Method and Description
PipelineResult.State	cancel() Cancels the pipeline execution.
DataflowPipelineOptions	getDataflowOptions()
java.lang.String	getJobId() Get the id of this job.
@Nullable org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline	getPipelineProto() Get the Runner API pipeline proto if available.
java.lang.String	getProjectId() Get the project this job exists in.
java.lang.String	getRegion() Get the region this job exists in.
DataflowPipelineJob	getReplacedByJob() Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.
PipelineResult.State	getState() Retrieves the current state of the pipeline execution.
protected @Nullable org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.BiMap<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String>	getTransformStepNames()
MetricResults	metrics() Returns the object to access metrics from the pipeline.
@Nullable PipelineResult.State	waitUntilFinish() Waits until the pipeline finishes and returns the final status.
@Nullable PipelineResult.State	waitUntilFinish(Duration duration) Waits until the pipeline finishes and returns the final status.
@Nullable PipelineResult.State	waitUntilFinish(Duration duration, MonitoringUtil.JobMessagesHandler messageHandler) Waits until the pipeline finishes and returns the final status.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

STATUS_BACKOFF_FACTORY

protected static final org.apache.beam.sdk.util.FluentBackoff STATUS_BACKOFF_FACTORY

Constructor Detail

DataflowPipelineJob

public DataflowPipelineJob(DataflowClient dataflowClient,
                           java.lang.String jobId,
                           DataflowPipelineOptions dataflowOptions,
                           java.util.Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames,
                           @Nullable org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipelineProto)

Constructs the job.

Parameters:

jobId - the job id

dataflowOptions - used to configure the client for the Dataflow Service

transformStepNames - a mapping from AppliedPTransforms to Step Names

pipelineProto - Runner API pipeline proto.

DataflowPipelineJob

public DataflowPipelineJob(DataflowClient dataflowClient,
                           java.lang.String jobId,
                           DataflowPipelineOptions dataflowOptions,
                           java.util.Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames)

Constructs the job.

Parameters:

jobId - the job id

dataflowOptions - used to configure the client for the Dataflow Service

transformStepNames - a mapping from AppliedPTransforms to Step Names

Method Detail

getJobId

public java.lang.String getJobId()

Get the id of this job.

getProjectId

public java.lang.String getProjectId()

Get the project this job exists in.

getDataflowOptions

public DataflowPipelineOptions getDataflowOptions()

getPipelineProto

public @Nullable org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline getPipelineProto()

Get the Runner API pipeline proto if available.

getRegion

public java.lang.String getRegion()

Get the region this job exists in.

getTransformStepNames

protected @Nullable org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.BiMap<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> getTransformStepNames()

getReplacedByJob

public DataflowPipelineJob getReplacedByJob()

Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.

Throws:

java.lang.IllegalStateException - if called before the job has terminated or if the job terminated but was not updated

waitUntilFinish

public @Nullable PipelineResult.State waitUntilFinish()

Description copied from interface: PipelineResult

Waits until the pipeline finishes and returns the final status.

Specified by:

waitUntilFinish in interface PipelineResult

Returns:

The final state of the pipeline.

waitUntilFinish

public @Nullable PipelineResult.State waitUntilFinish(Duration duration)

Description copied from interface: PipelineResult

Waits until the pipeline finishes and returns the final status. It times out after the given duration.

Specified by:

waitUntilFinish in interface PipelineResult

Parameters:

duration - The time to wait for the pipeline to finish. Provide a value less than 1 ms for an infinite wait.

Returns:

The final state of the pipeline or null on timeout.

waitUntilFinish

public @Nullable PipelineResult.State waitUntilFinish(Duration duration,
                                                      MonitoringUtil.JobMessagesHandler messageHandler)
                                               throws java.io.IOException,
                                                      java.lang.InterruptedException

Waits until the pipeline finishes and returns the final status.

Parameters:

duration - The time to wait for the job to finish. Provide a value less than 1 ms for an infinite wait.

messageHandler - If non null this handler will be invoked for each batch of messages received.

Returns:

The final state of the job or null on timeout or if the thread is interrupted.

Throws:

java.io.IOException - If there is a persistent problem getting job information.

java.lang.InterruptedException

cancel

public PipelineResult.State cancel()
                            throws java.io.IOException

Description copied from interface: PipelineResult

Cancels the pipeline execution.

Specified by:

cancel in interface PipelineResult

Throws:

java.io.IOException - if there is a problem executing the cancel request.

getState

public PipelineResult.State getState()

Description copied from interface: PipelineResult

Retrieves the current state of the pipeline execution.

Specified by:

getState in interface PipelineResult

Returns:

the PipelineResult.State representing the state of this pipeline.

metrics

public MetricResults metrics()

Description copied from interface: PipelineResult

Returns the object to access metrics from the pipeline.

Specified by:

metrics in interface PipelineResult