Package org.apache.beam.runners.jobsubmission

Class InMemoryJobService

java.lang.Object
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase
org.apache.beam.runners.jobsubmission.InMemoryJobService
All Implemented Interfaces:
AutoCloseable, org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService, FnService, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.BindableService
public class InMemoryJobService extends org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase implements FnService
A InMemoryJobService that prepares and runs jobs on behalf of a client using a JobInvoker.

Job management is handled in-memory rather than any persistent storage, running the risk of leaking jobs if the InMemoryJobService crashes.

TODO: replace in-memory job management state with persistent solution.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    DEFAULT_MAX_INVOCATION_HISTORY
    The default maximum number of completed invocations to keep.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    cancel(org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse> responseObserver)
     
    void
    close()
    .
    static InMemoryJobService
    create(GrpcFnServer<ArtifactStagingService> stagingService, Function<String,String> stagingServiceTokenProvider, ThrowingConsumer<Exception,String> cleanupJobFn, JobInvoker invoker)
    Creates an InMemoryJobService.
    static InMemoryJobService
    create(GrpcFnServer<ArtifactStagingService> stagingService, Function<String,String> stagingServiceTokenProvider, ThrowingConsumer<Exception,String> cleanupJobFn, JobInvoker invoker, int maxInvocationHistory)
    Creates an InMemoryJobService.
    void
    describePipelineOptions(org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsResponse> responseObserver)
     
    void
    getJobMetrics(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobMetricsRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobMetricsResponse> responseObserver)
     
    void
    getJobs(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsResponse> responseObserver)
     
    void
    getMessageStream(org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse> responseObserver)
     
    void
    getPipeline(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineResponse> responseObserver)
     
    void
    getState(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent> responseObserver)
     
    void
    getStateStream(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent> responseObserver)
     
    void
    prepare(org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse> responseObserver)
     
    void
    run(org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse> responseObserver)
     

    Methods inherited from class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

    bindService

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.beam.vendor.grpc.v1p69p0.io.grpc.BindableService

    bindService

    Methods inherited from interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    drain

  • Field Details

    • DEFAULT_MAX_INVOCATION_HISTORY

      public static final int DEFAULT_MAX_INVOCATION_HISTORY
      The default maximum number of completed invocations to keep.
      See Also:

  • Method Details

    • create

      public static InMemoryJobService create(GrpcFnServer<ArtifactStagingService> stagingService, Function<String,String> stagingServiceTokenProvider, ThrowingConsumer<Exception,String> cleanupJobFn, JobInvoker invoker)
      Creates an InMemoryJobService.
      Parameters:
      stagingService - The staging service.
      stagingServiceTokenProvider - Function mapping a preparationId to a staging service token.
      cleanupJobFn - A cleanup function to run, parameterized with the staging token of a job.
      invoker - A JobInvoker which creates the jobs.
      Returns:
      A new InMemoryJobService.

    • create

      public static InMemoryJobService create(GrpcFnServer<ArtifactStagingService> stagingService, Function<String,String> stagingServiceTokenProvider, ThrowingConsumer<Exception,String> cleanupJobFn, JobInvoker invoker, int maxInvocationHistory)
      Creates an InMemoryJobService.
      Parameters:
      stagingService - The staging service.
      stagingServiceTokenProvider - Function mapping a preparationId to a staging service token.
      cleanupJobFn - A cleanup function to run, parameterized with the staging token of a job.
      invoker - A JobInvoker which creates the jobs.
      maxInvocationHistory - The maximum number of completed invocations to keep.
      Returns:
      A new InMemoryJobService.

    • prepare

      public void prepare(org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse> responseObserver)
      Specified by:
      prepare in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • run

      public void run(org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse> responseObserver)
      Specified by:
      run in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • getJobs

      public void getJobs(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsResponse> responseObserver)
      Specified by:
      getJobs in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • getState

      public void getState(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent> responseObserver)
      Specified by:
      getState in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • getPipeline

      public void getPipeline(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineResponse> responseObserver)
      Specified by:
      getPipeline in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • cancel

      public void cancel(org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse> responseObserver)
      Specified by:
      cancel in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • getStateStream

      public void getStateStream(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent> responseObserver)
      Specified by:
      getStateStream in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • getMessageStream

      public void getMessageStream(org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse> responseObserver)
      Specified by:
      getMessageStream in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • getJobMetrics

      public void getJobMetrics(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobMetricsRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobMetricsResponse> responseObserver)
      Specified by:
      getJobMetrics in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • describePipelineOptions

      public void describePipelineOptions(org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsRequest request, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsResponse> responseObserver)
      Specified by:
      describePipelineOptions in interface org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.AsyncService

    • close

      public void close() throws Exception
      Description copied from interface: FnService
      .

      There should be no more calls to any service method by the time a call to FnService.close() begins. Specifically, this means that a Server that this service is bound to should have completed a call to the Server.shutdown() method, and all future incoming calls will be rejected.

      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface FnService
      Throws:
      Exception