org.apache.beam.runners.fnexecution.artifact

Class ArtifactStagingService

java.lang.Object

org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase

org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService

All Implemented Interfaces:

java.lang.AutoCloseable, FnService, org.apache.beam.vendor.grpc.v1p26p0.io.grpc.BindableService

public class ArtifactStagingService
extends org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase
implements FnService

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ArtifactStagingService.ArtifactDestination A pairing of a newly created artifact type and an ouptut stream that will be readable at that type.
static interface	ArtifactStagingService.ArtifactDestinationProvider Provides a concrete location to which artifacts can be staged on retrieval.

Constructor Summary

Constructors

Constructor and Description
ArtifactStagingService(ArtifactStagingService.ArtifactDestinationProvider destinationProvider)

Method Summary

Modifier and Type	Method and Description
static ArtifactStagingService.ArtifactDestinationProvider	beamFilesystemArtifactDestinationProvider(java.lang.String root) An ArtifactDestinationProvider that places new artifacts as files in a Beam filesystem.
void	close() .
java.util.Map<java.lang.String,java.util.List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>>	getStagedArtifacts(java.lang.String stagingToken) Returns the rewritten artifacts associated with this job, keyed by environment.
static void	offer(ArtifactRetrievalService retrievalService, org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingService, java.lang.String stagingToken) Lazily stages artifacts by letting an ArtifactStagingService resolve and request artifacts.
void	registerJob(java.lang.String stagingToken, java.util.Map<java.lang.String,java.util.List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> artifacts) Registers a set of artifacts to be staged with this service.
void	removeStagedArtifacts(java.lang.String stagingToken)
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactResponseWrapper>	reverseArtifactRetrievalService(org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactRequestWrapper> responseObserver)

Methods inherited from class org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase

bindService

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.beam.vendor.grpc.v1p26p0.io.grpc.BindableService

bindService

Constructor Detail

ArtifactStagingService

public ArtifactStagingService(ArtifactStagingService.ArtifactDestinationProvider destinationProvider)

Method Detail

registerJob

public void registerJob(java.lang.String stagingToken,
                        java.util.Map<java.lang.String,java.util.List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> artifacts)

Registers a set of artifacts to be staged with this service.

A client (e.g. a Beam SDK) is expected to connect to this service with the given staging token and offer resolution and retrieval of this set of artifacts.

Parameters:

stagingToken - a staging token for this job

artifacts - all artifacts to stage, keyed by environment

getStagedArtifacts

public java.util.Map<java.lang.String,java.util.List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> getStagedArtifacts(java.lang.String stagingToken)

Returns the rewritten artifacts associated with this job, keyed by environment.

This should be called after the client has finished offering artifacts.

Parameters:

stagingToken - a staging token for this job

removeStagedArtifacts

public void removeStagedArtifacts(java.lang.String stagingToken)
                           throws java.io.IOException

Throws:

java.io.IOException

beamFilesystemArtifactDestinationProvider

public static ArtifactStagingService.ArtifactDestinationProvider beamFilesystemArtifactDestinationProvider(java.lang.String root)

An ArtifactDestinationProvider that places new artifacts as files in a Beam filesystem.

Parameters:

root - the directory in which to place all artifacts

reverseArtifactRetrievalService

public org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactResponseWrapper> reverseArtifactRetrievalService(org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactRequestWrapper> responseObserver)

Overrides:

reverseArtifactRetrievalService in class org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase

close

public void close()
           throws java.lang.Exception

Description copied from interface: FnService

.

There should be no more calls to any service method by the time a call to FnService.close() begins. Specifically, this means that a Server that this service is bound to should have completed a call to the Server.shutdown() method, and all future incoming calls will be rejected.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface FnService

Throws:

java.lang.Exception

offer

public static void offer(ArtifactRetrievalService retrievalService,
                         org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingService,
                         java.lang.String stagingToken)
                  throws java.util.concurrent.ExecutionException,
                         java.lang.InterruptedException

Lazily stages artifacts by letting an ArtifactStagingService resolve and request artifacts.

Parameters:

retrievalService - an ArtifactRetrievalService used to resolve and retrieve artifacts

stagingService - an ArtifactStagingService stub which will request artifacts

stagingToken - the staging token of the job whose artifacts will be retrieved

Throws:

java.lang.InterruptedException

java.io.IOException

java.util.concurrent.ExecutionException