org.apache.beam.runners.flink

Class ArtifactSourcePool

java.lang.Object

org.apache.beam.runners.flink.ArtifactSourcePool

All Implemented Interfaces:

ArtifactSource

@ThreadSafe
public class ArtifactSourcePool
extends java.lang.Object
implements ArtifactSource

A pool of ArtifactSources that can be used as a single source. At least one source must be registered before artifacts can be requested from it.

Artifact pooling is required for Flink operators that use the DistributedCache for artifact distribution. This is because distributed caches (along with other runtime context) are scoped to operator lifetimes but the artifact retrieval service must outlive the any remote environments it serves. Remote environments cannot be shared between jobs and are thus job-scoped.

Because of the peculiarities of artifact pooling and Flink, this class is packaged with the Flink runner rather than as a core fn-execution utility.

Method Summary

Modifier and Type	Method and Description
java.lang.AutoCloseable	addToPool(ArtifactSource artifactSource) Adds a new cache to the pool.
void	getArtifact(java.lang.String name, io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk> responseObserver) Get an artifact by its name.
org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest	getManifest() Get the artifact manifest available from this source.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

addToPool

public java.lang.AutoCloseable addToPool(ArtifactSource artifactSource)

Adds a new cache to the pool. When the returned AutoCloseable is closed, the given cache will be removed from the pool.

getManifest

public org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest getManifest()
                                                                        throws java.io.IOException

Description copied from interface: ArtifactSource

Get the artifact manifest available from this source.

Specified by:

getManifest in interface ArtifactSource

Throws:

java.io.IOException

getArtifact

public void getArtifact(java.lang.String name,
                        io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk> responseObserver)

Description copied from interface: ArtifactSource

Get an artifact by its name.

Specified by:

getArtifact in interface ArtifactSource