org.apache.beam.runners.fnexecution.jobsubmission

Class InMemoryJobService

java.lang.Object

org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService

All Implemented Interfaces:

java.lang.AutoCloseable, FnService, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.BindableService

public class InMemoryJobService
extends org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase
implements FnService

A InMemoryJobService that prepares and runs jobs on behalf of a client using a JobInvoker.

Job management is handled in-memory rather than any persistent storage, running the risk of leaking jobs if the InMemoryJobService crashes.

TODO: replace in-memory job management state with persistent solution.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAX_INVOCATION_HISTORY The default maximum number of completed invocations to keep.

Method Summary

Modifier and Type	Method and Description
void	cancel(org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse> responseObserver)
void	close() .
static InMemoryJobService	create(org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor stagingServiceDescriptor, java.util.function.Function<java.lang.String,java.lang.String> stagingServiceTokenProvider, ThrowingConsumer<java.lang.Exception,java.lang.String> cleanupJobFn, JobInvoker invoker) Creates an InMemoryJobService.
static InMemoryJobService	create(org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor stagingServiceDescriptor, java.util.function.Function<java.lang.String,java.lang.String> stagingServiceTokenProvider, ThrowingConsumer<java.lang.Exception,java.lang.String> cleanupJobFn, JobInvoker invoker, int maxInvocationHistory) Creates an InMemoryJobService.
void	describePipelineOptions(org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsResponse> responseObserver)
void	getJobMetrics(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobMetricsRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobMetricsResponse> responseObserver)
void	getJobs(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsResponse> responseObserver)
void	getMessageStream(org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse> responseObserver)
void	getPipeline(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineResponse> responseObserver)
void	getState(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent> responseObserver)
void	getStateStream(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent> responseObserver)
void	prepare(org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse> responseObserver)
void	run(org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest request, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse> responseObserver)

Methods inherited from class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

bindService

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.beam.vendor.grpc.v1p26p0.io.grpc.BindableService

bindService

Field Detail

DEFAULT_MAX_INVOCATION_HISTORY

public static final int DEFAULT_MAX_INVOCATION_HISTORY

The default maximum number of completed invocations to keep.

See Also:

Constant Field Values

Method Detail

create

public static InMemoryJobService create(org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor stagingServiceDescriptor,
                                        java.util.function.Function<java.lang.String,java.lang.String> stagingServiceTokenProvider,
                                        ThrowingConsumer<java.lang.Exception,java.lang.String> cleanupJobFn,
                                        JobInvoker invoker)

Creates an InMemoryJobService.

Parameters:

stagingServiceDescriptor - Endpoint for the staging service.

stagingServiceTokenProvider - Function mapping a preparationId to a staging service token.

cleanupJobFn - A cleanup function to run, parameterized with the staging token of a job.

invoker - A JobInvoker which creates the jobs.

Returns:

A new InMemoryJobService.

create

public static InMemoryJobService create(org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor stagingServiceDescriptor,
                                        java.util.function.Function<java.lang.String,java.lang.String> stagingServiceTokenProvider,
                                        ThrowingConsumer<java.lang.Exception,java.lang.String> cleanupJobFn,
                                        JobInvoker invoker,
                                        int maxInvocationHistory)

Creates an InMemoryJobService.

Parameters:

stagingServiceDescriptor - The endpoint for the staging service.

stagingServiceTokenProvider - Function mapping a preparationId to a staging service token.

cleanupJobFn - A cleanup function to run, parameterized with the staging token of a job.

invoker - A JobInvoker which creates the jobs.

maxInvocationHistory - The maximum number of completed invocations to keep.

Returns:

A new InMemoryJobService.

prepare

public void prepare(org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest request,
                    org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse> responseObserver)

Overrides:

prepare in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

run

public void run(org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest request,
                org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse> responseObserver)

Overrides:

run in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

getJobs

public void getJobs(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsRequest request,
                    org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsResponse> responseObserver)

Overrides:

getJobs in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

getState

public void getState(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest request,
                     org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent> responseObserver)

Overrides:

getState in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

getPipeline

public void getPipeline(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineRequest request,
                        org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineResponse> responseObserver)

Overrides:

getPipeline in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

cancel

public void cancel(org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest request,
                   org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse> responseObserver)

Overrides:

cancel in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

getStateStream

public void getStateStream(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest request,
                           org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobStateEvent> responseObserver)

Overrides:

getStateStream in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

getMessageStream

public void getMessageStream(org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest request,
                             org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse> responseObserver)

Overrides:

getMessageStream in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

getJobMetrics

public void getJobMetrics(org.apache.beam.model.jobmanagement.v1.JobApi.GetJobMetricsRequest request,
                          org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.GetJobMetricsResponse> responseObserver)

Overrides:

getJobMetrics in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

describePipelineOptions

public void describePipelineOptions(org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsRequest request,
                                    org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsResponse> responseObserver)

Overrides:

describePipelineOptions in class org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase

close

public void close()
           throws java.lang.Exception

Description copied from interface: FnService

.

There should be no more calls to any service method by the time a call to FnService.close() begins. Specifically, this means that a Server that this service is bound to should have completed a call to the Server.shutdown() method, and all future incoming calls will be rejected.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface FnService

Throws:

java.lang.Exception