Package org.apache.beam.runners.fnexecution.artifact

Class ArtifactStagingService

java.lang.Object

org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase

org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService

All Implemented Interfaces:

AutoCloseable, org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.AsyncService, FnService, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.BindableService

public class ArtifactStagingService
extends org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase
implements FnService

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ArtifactStagingService.ArtifactDestination	A pairing of a newly created artifact type and an output stream that will be readable at that type.
static interface	ArtifactStagingService.ArtifactDestinationProvider	Provides a concrete location to which artifacts can be staged on retrieval.

Constructor Summary

Constructors

Constructor	Description
ArtifactStagingService(ArtifactStagingService.ArtifactDestinationProvider destinationProvider)

Method Summary

Modifier and Type	Method	Description
static ArtifactStagingService.ArtifactDestinationProvider	beamFilesystemArtifactDestinationProvider(String root)	An ArtifactDestinationProvider that places new artifacts as files in a Beam filesystem.
void	close()	.
Map<String,List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>>	getStagedArtifacts(String stagingToken)	Returns the rewritten artifacts associated with this job, keyed by environment.
static void	offer(ArtifactRetrievalService retrievalService, org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingService, String stagingToken)	Lazily stages artifacts by letting an ArtifactStagingService resolve and request artifacts.
void	registerJob(String stagingToken, Map<String,List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> artifacts)	Registers a set of artifacts to be staged with this service.
void	removeStagedArtifacts(String stagingToken)
org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactResponseWrapper>	reverseArtifactRetrievalService(org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactRequestWrapper> responseObserver)

Methods inherited from class org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase

bindService

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.beam.vendor.grpc.v1p69p0.io.grpc.BindableService

bindService

Constructor Details

ArtifactStagingService

public ArtifactStagingService(ArtifactStagingService.ArtifactDestinationProvider destinationProvider)

Method Details

registerJob

public void registerJob(String stagingToken,
 Map<String,List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> artifacts)

Registers a set of artifacts to be staged with this service.

A client (e.g. a Beam SDK) is expected to connect to this service with the given staging token and offer resolution and retrieval of this set of artifacts.

Parameters:

stagingToken - a staging token for this job

artifacts - all artifacts to stage, keyed by environment

getStagedArtifacts

public Map<String,List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> getStagedArtifacts(String stagingToken)

Returns the rewritten artifacts associated with this job, keyed by environment.

This should be called after the client has finished offering artifacts.

Parameters:

stagingToken - a staging token for this job

removeStagedArtifacts

public void removeStagedArtifacts(String stagingToken)
                           throws IOException

Throws:

IOException

beamFilesystemArtifactDestinationProvider

public static ArtifactStagingService.ArtifactDestinationProvider beamFilesystemArtifactDestinationProvider(String root)

An ArtifactDestinationProvider that places new artifacts as files in a Beam filesystem.

Parameters:

root - the directory in which to place all artifacts

reverseArtifactRetrievalService

public org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactResponseWrapper> reverseArtifactRetrievalService(org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactRequestWrapper> responseObserver)

Specified by:

reverseArtifactRetrievalService in interface org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.AsyncService

close

public void close()
           throws Exception

Description copied from interface: FnService

.

There should be no more calls to any service method by the time a call to FnService.close() begins. Specifically, this means that a Server that this service is bound to should have completed a call to the Server.shutdown() method, and all future incoming calls will be rejected.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface FnService

Throws:

Exception

offer

public static void offer(ArtifactRetrievalService retrievalService,
 org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingService,
 String stagingToken)
                  throws ExecutionException,
InterruptedException

Lazily stages artifacts by letting an ArtifactStagingService resolve and request artifacts.

Parameters:

retrievalService - an ArtifactRetrievalService used to resolve and retrieve artifacts

stagingService - an ArtifactStagingService stub which will request artifacts

stagingToken - the staging token of the job whose artifacts will be retrieved

Throws:

InterruptedException

IOException

ExecutionException