org.apache.beam.sdk.extensions.gcp.util

Class GcsUtil

java.lang.Object

org.apache.beam.sdk.extensions.gcp.util.GcsUtil

public class GcsUtil
extends java.lang.Object

Provides operations on GCS.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GcsUtil.CreateOptions
static class	GcsUtil.GcsCountersOptions
static class	GcsUtil.GcsUtilFactory This is a DefaultValueFactory able to create a GcsUtil using any transport flags specified on the PipelineOptions.
static class	GcsUtil.StorageObjectOrIOException A class that holds either a StorageObject or an IOException.

Method Summary

Modifier and Type	Method and Description
boolean	bucketAccessible(GcsPath path) Returns whether the GCS bucket exists and is accessible.
long	bucketOwner(GcsPath path) Returns the project number of the project which owns this bucket.
void	copy(java.lang.Iterable<java.lang.String> srcFilenames, java.lang.Iterable<java.lang.String> destFilenames)
java.nio.channels.WritableByteChannel	create(GcsPath path, GcsUtil.CreateOptions options) Creates an object in GCS and prepares for uploading its contents.
java.nio.channels.WritableByteChannel	create(GcsPath path, java.lang.String type) Deprecated. Use create(GcsPath, CreateOptions) instead.
java.nio.channels.WritableByteChannel	create(GcsPath path, java.lang.String type, java.lang.Integer uploadBufferSizeBytes) Deprecated. Use create(GcsPath, CreateOptions) instead.
void	createBucket(java.lang.String projectId, Bucket bucket) Creates a Bucket under the specified project in Cloud Storage or propagates an exception.
java.util.List<GcsPath>	expand(GcsPath gcsPattern) Expands a pattern into matched paths.
long	fileSize(GcsPath path) Returns the file size from GCS or throws FileNotFoundException if the resource does not exist.
@Nullable Bucket	getBucket(GcsPath path) Get the Bucket from Cloud Storage path or propagates an exception.
static java.lang.String	getNonWildcardPrefix(java.lang.String globExp) Returns the prefix portion of the glob that doesn't contain wildcards.
StorageObject	getObject(GcsPath gcsPath) Returns the StorageObject for the given GcsPath.
java.util.List<GcsUtil.StorageObjectOrIOException>	getObjects(java.util.List<GcsPath> gcsPaths) Returns StorageObjectOrIOExceptions for the given GcsPaths.
static boolean	isWildcard(GcsPath spec) Returns true if the given spec contains wildcard.
Objects	listObjects(java.lang.String bucket, java.lang.String prefix, @Nullable java.lang.String pageToken)
Objects	listObjects(java.lang.String bucket, java.lang.String prefix, @Nullable java.lang.String pageToken, @Nullable java.lang.String delimiter) Lists Objects given the bucket, prefix, pageToken.
java.nio.channels.SeekableByteChannel	open(GcsPath path) Opens an object in GCS.
void	remove(java.util.Collection<java.lang.String> filenames)
void	removeBucket(Bucket bucket) Remove an empty Bucket in Cloud Storage or propagates an exception.
void	rename(java.lang.Iterable<java.lang.String> srcFilenames, java.lang.Iterable<java.lang.String> destFilenames, MoveOptions... moveOptions)
protected void	setBatchRequestSupplier(java.util.function.Supplier<org.apache.beam.sdk.extensions.gcp.util.GcsUtil.BatchInterface> supplier)
protected void	setStorageClient(Storage storageClient)
void	verifyBucketAccessible(GcsPath path) Checks whether the GCS bucket exists.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getNonWildcardPrefix

public static java.lang.String getNonWildcardPrefix(java.lang.String globExp)

Returns the prefix portion of the glob that doesn't contain wildcards.

isWildcard

public static boolean isWildcard(GcsPath spec)

Returns true if the given spec contains wildcard.

setStorageClient

protected void setStorageClient(Storage storageClient)

setBatchRequestSupplier

protected void setBatchRequestSupplier(java.util.function.Supplier<org.apache.beam.sdk.extensions.gcp.util.GcsUtil.BatchInterface> supplier)

expand

public java.util.List<GcsPath> expand(GcsPath gcsPattern)
                               throws java.io.IOException

Expands a pattern into matched paths. The pattern path may contain globs, which are expanded in the result. For patterns that only match a single object, we ensure that the object exists.

Throws:

java.io.IOException

fileSize

public long fileSize(GcsPath path)
              throws java.io.IOException

Returns the file size from GCS or throws FileNotFoundException if the resource does not exist.

Throws:

java.io.IOException

getObject

public StorageObject getObject(GcsPath gcsPath)
                        throws java.io.IOException

Returns the StorageObject for the given GcsPath.

Throws:

java.io.IOException

getObjects

public java.util.List<GcsUtil.StorageObjectOrIOException> getObjects(java.util.List<GcsPath> gcsPaths)
                                                              throws java.io.IOException

Returns StorageObjectOrIOExceptions for the given GcsPaths.

Throws:

java.io.IOException

listObjects

public Objects listObjects(java.lang.String bucket,
                           java.lang.String prefix,
                           @Nullable java.lang.String pageToken)
                    throws java.io.IOException

Throws:

java.io.IOException

listObjects

public Objects listObjects(java.lang.String bucket,
                           java.lang.String prefix,
                           @Nullable java.lang.String pageToken,
                           @Nullable java.lang.String delimiter)
                    throws java.io.IOException

Lists Objects given the bucket, prefix, pageToken.

For more details, see https://cloud.google.com/storage/docs/json_api/v1/objects/list.

Throws:

java.io.IOException

open

public java.nio.channels.SeekableByteChannel open(GcsPath path)
                                           throws java.io.IOException

Opens an object in GCS.

Returns a SeekableByteChannel that provides access to data in the bucket.

Parameters:

path - the GCS filename to read from

Returns:

a SeekableByteChannel that can read the object data

Throws:

java.io.IOException

create

@Deprecated
public java.nio.channels.WritableByteChannel create(GcsPath path,
                                                                java.lang.String type)
                                                         throws java.io.IOException

Deprecated. Use create(GcsPath, CreateOptions) instead.

Throws:

java.io.IOException

create

@Deprecated
public java.nio.channels.WritableByteChannel create(GcsPath path,
                                                                java.lang.String type,
                                                                java.lang.Integer uploadBufferSizeBytes)
                                                         throws java.io.IOException

Deprecated. Use create(GcsPath, CreateOptions) instead.

Throws:

java.io.IOException

create

public java.nio.channels.WritableByteChannel create(GcsPath path,
                                                    GcsUtil.CreateOptions options)
                                             throws java.io.IOException

Creates an object in GCS and prepares for uploading its contents.

Parameters:

path - the GCS file to write to

options - to be used for creating and configuring file upload

Returns:

a WritableByteChannel that can be used to write data to the object.

Throws:

java.io.IOException

verifyBucketAccessible

public void verifyBucketAccessible(GcsPath path)
                            throws java.io.IOException

Checks whether the GCS bucket exists. Similar to bucketAccessible(GcsPath), but throws exception if the bucket is inaccessible due to permissions or does not exist.

Throws:

java.io.IOException

bucketAccessible

public boolean bucketAccessible(GcsPath path)
                         throws java.io.IOException

Returns whether the GCS bucket exists and is accessible.

Throws:

java.io.IOException

bucketOwner

public long bucketOwner(GcsPath path)
                 throws java.io.IOException

Returns the project number of the project which owns this bucket. If the bucket exists, it must be accessible otherwise the permissions exception will be propagated. If the bucket does not exist, an exception will be thrown.

Throws:

java.io.IOException

createBucket

public void createBucket(java.lang.String projectId,
                         Bucket bucket)
                  throws java.io.IOException

Creates a Bucket under the specified project in Cloud Storage or propagates an exception.

Throws:

java.io.IOException

getBucket

public @Nullable Bucket getBucket(GcsPath path)
                           throws java.io.IOException

Get the Bucket from Cloud Storage path or propagates an exception.

Throws:

java.io.IOException

removeBucket

public void removeBucket(Bucket bucket)
                  throws java.io.IOException

Remove an empty Bucket in Cloud Storage or propagates an exception.

Throws:

java.io.IOException

copy

public void copy(java.lang.Iterable<java.lang.String> srcFilenames,
                 java.lang.Iterable<java.lang.String> destFilenames)
          throws java.io.IOException

Throws:

java.io.IOException

rename

public void rename(java.lang.Iterable<java.lang.String> srcFilenames,
                   java.lang.Iterable<java.lang.String> destFilenames,
                   MoveOptions... moveOptions)
            throws java.io.IOException

Throws:

java.io.IOException

remove

public void remove(java.util.Collection<java.lang.String> filenames)
            throws java.io.IOException

Throws:

java.io.IOException