Package org.apache.beam.runners.dataflow

Class DataflowPipelineJob

java.lang.Object
org.apache.beam.runners.dataflow.DataflowPipelineJob
All Implemented Interfaces:
PipelineResult
Direct Known Subclasses:
DataflowTemplateJob
public class DataflowPipelineJob extends Object implements PipelineResult
A DataflowPipelineJob represents a job submitted to Dataflow using DataflowRunner.

  • Field Details

    • STATUS_BACKOFF_FACTORY

      protected static final org.apache.beam.sdk.util.FluentBackoff STATUS_BACKOFF_FACTORY

  • Constructor Details

    • DataflowPipelineJob

      public DataflowPipelineJob(DataflowClient dataflowClient, String jobId, DataflowPipelineOptions dataflowOptions, Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,String> transformStepNames, @Nullable org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipelineProto)
      Constructs the job.
      Parameters:
      jobId - the job id
      dataflowOptions - used to configure the client for the Dataflow Service
      transformStepNames - a mapping from AppliedPTransforms to Step Names
      pipelineProto - Runner API pipeline proto.

    • DataflowPipelineJob

      public DataflowPipelineJob(DataflowClient dataflowClient, String jobId, DataflowPipelineOptions dataflowOptions, Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,String> transformStepNames)
      Constructs the job.
      Parameters:
      jobId - the job id
      dataflowOptions - used to configure the client for the Dataflow Service
      transformStepNames - a mapping from AppliedPTransforms to Step Names

  • Method Details

    • getJobId

      public String getJobId()
      Get the id of this job.

    • getProjectId

      public String getProjectId()
      Get the project this job exists in.

    • getDataflowOptions

      public DataflowPipelineOptions getDataflowOptions()

    • getPipelineProto

      public @Nullable org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline getPipelineProto()
      Get the Runner API pipeline proto if available.

    • getRegion

      public String getRegion()
      Get the region this job exists in.

    • getTransformStepNames

      protected @Nullable org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.BiMap<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,String> getTransformStepNames()

    • getReplacedByJob

      public DataflowPipelineJob getReplacedByJob()
      Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.
      Throws:
      IllegalStateException - if called before the job has terminated or if the job terminated but was not updated

    • waitUntilFinish

      public @Nullable PipelineResult.State waitUntilFinish()
      Description copied from interface: PipelineResult
      Waits until the pipeline finishes and returns the final status.
      Specified by:
      waitUntilFinish in interface PipelineResult
      Returns:
      The final state of the pipeline.

    • waitUntilFinish

      public @Nullable PipelineResult.State waitUntilFinish(Duration duration)
      Description copied from interface: PipelineResult
      Waits until the pipeline finishes and returns the final status. It times out after the given duration.
      Specified by:
      waitUntilFinish in interface PipelineResult
      Parameters:
      duration - The time to wait for the pipeline to finish. Provide a value less than 1 ms for an infinite wait.
      Returns:
      The final state of the pipeline or null on timeout.

    • waitUntilFinish

      public @Nullable PipelineResult.State waitUntilFinish(Duration duration, MonitoringUtil.JobMessagesHandler messageHandler) throws IOException, InterruptedException
      Waits until the pipeline finishes and returns the final status.
      Parameters:
      duration - The time to wait for the job to finish. Provide a value less than 1 ms for an infinite wait.
      messageHandler - If non null this handler will be invoked for each batch of messages received.
      Returns:
      The final state of the job or null on timeout or if the thread is interrupted.
      Throws:
      IOException - If there is a persistent problem getting job information.
      InterruptedException

    • cancel

      public PipelineResult.State cancel() throws IOException
      Description copied from interface: PipelineResult
      Cancels the pipeline execution.
      Specified by:
      cancel in interface PipelineResult
      Throws:
      IOException - if there is a problem executing the cancel request.

    • getState

      public PipelineResult.State getState()
      Description copied from interface: PipelineResult
      Retrieves the current state of the pipeline execution.
      Specified by:
      getState in interface PipelineResult
      Returns:
      the PipelineResult.State representing the state of this pipeline.

    • metrics

      public MetricResults metrics()
      Description copied from interface: PipelineResult
      Returns the object to access metrics from the pipeline.
      Specified by:
      metrics in interface PipelineResult