Package org.apache.beam.sdk.extensions.gcp.util

Class GcsUtil

java.lang.Object

org.apache.beam.sdk.extensions.gcp.util.GcsUtil

public class GcsUtil
extends Object

Provides operations on GCS.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	GcsUtil.CreateOptions
static class	GcsUtil.GcsCountersOptions
static class	GcsUtil.GcsReadOptionsFactory
static class	GcsUtil.GcsUtilFactory	This is a DefaultValueFactory able to create a GcsUtil using any transport flags specified on the PipelineOptions.
static class	GcsUtil.StorageObjectOrIOException	A class that holds either a StorageObject or an IOException.

Method Summary

Modifier and Type	Method	Description
boolean	bucketAccessible(GcsPath path)	Returns whether the GCS bucket exists and is accessible.
long	bucketOwner(GcsPath path)	Returns the project number of the project which owns this bucket.
void	copy(Iterable<String> srcFilenames, Iterable<String> destFilenames)
WritableByteChannel	create(GcsPath path, String type)	Deprecated. Use create(GcsPath, CreateOptions) instead.
WritableByteChannel	create(GcsPath path, String type, Integer uploadBufferSizeBytes)	Deprecated. Use create(GcsPath, CreateOptions) instead.
WritableByteChannel	create(GcsPath path, GcsUtil.CreateOptions options)	Creates an object in GCS and prepares for uploading its contents.
void	createBucket(String projectId, Bucket bucket)	Creates a Bucket under the specified project in Cloud Storage or propagates an exception.
List<GcsPath>	expand(GcsPath gcsPattern)	Expands a pattern into matched paths.
long	fileSize(GcsPath path)	Returns the file size from GCS or throws FileNotFoundException if the resource does not exist.
@Nullable Bucket	getBucket(GcsPath path)	Get the Bucket from Cloud Storage path or propagates an exception.
static String	getNonWildcardPrefix(String globExp)	Returns the prefix portion of the glob that doesn't contain wildcards.
StorageObject	getObject(GcsPath gcsPath)	Returns the StorageObject for the given GcsPath.
List<GcsUtil.StorageObjectOrIOException>	getObjects(List<GcsPath> gcsPaths)	Returns StorageObjectOrIOExceptions for the given GcsPaths.
static boolean	isWildcard(GcsPath spec)	Returns true if the given spec contains wildcard.
Objects	listObjects(String bucket, String prefix, @Nullable String pageToken)
Objects	listObjects(String bucket, String prefix, @Nullable String pageToken, @Nullable String delimiter)	Lists Objects given the bucket, prefix, pageToken.
SeekableByteChannel	open(GcsPath path)	Opens an object in GCS.
void	remove(Collection<String> filenames)
void	removeBucket(Bucket bucket)	Remove an empty Bucket in Cloud Storage or propagates an exception.
void	rename(Iterable<String> srcFilenames, Iterable<String> destFilenames, MoveOptions... moveOptions)
protected void	setBatchRequestSupplier(Supplier<org.apache.beam.sdk.extensions.gcp.util.GcsUtil.BatchInterface> supplier)
protected void	setStorageClient(Storage storageClient)
void	verifyBucketAccessible(GcsPath path)	Checks whether the GCS bucket exists.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

getNonWildcardPrefix

public static String getNonWildcardPrefix(String globExp)

Returns the prefix portion of the glob that doesn't contain wildcards.

isWildcard

public static boolean isWildcard(GcsPath spec)

Returns true if the given spec contains wildcard.

setStorageClient

protected void setStorageClient(Storage storageClient)

setBatchRequestSupplier

protected void setBatchRequestSupplier(Supplier<org.apache.beam.sdk.extensions.gcp.util.GcsUtil.BatchInterface> supplier)

expand

public List<GcsPath> expand(GcsPath gcsPattern)
                     throws IOException

Expands a pattern into matched paths. The pattern path may contain globs, which are expanded in the result. For patterns that only match a single object, we ensure that the object exists.

Throws:

IOException

fileSize

public long fileSize(GcsPath path)
              throws IOException

Returns the file size from GCS or throws FileNotFoundException if the resource does not exist.

Throws:

IOException

getObject

public StorageObject getObject(GcsPath gcsPath)
                        throws IOException

Returns the StorageObject for the given GcsPath.

Throws:

IOException

getObjects

public List<GcsUtil.StorageObjectOrIOException> getObjects(List<GcsPath> gcsPaths)
                                                    throws IOException

Returns StorageObjectOrIOExceptions for the given GcsPaths.

Throws:

IOException

listObjects

public Objects listObjects(String bucket,
 String prefix,
 @Nullable String pageToken)
                    throws IOException

Throws:

IOException

listObjects

public Objects listObjects(String bucket,
 String prefix,
 @Nullable String pageToken,
 @Nullable String delimiter)
                    throws IOException

Lists Objects given the bucket, prefix, pageToken.

For more details, see https://cloud.google.com/storage/docs/json_api/v1/objects/list.

Throws:

IOException

open

public SeekableByteChannel open(GcsPath path)
                         throws IOException

Opens an object in GCS.

Returns a SeekableByteChannel that provides access to data in the bucket.

Parameters:

path - the GCS filename to read from

Returns:

a SeekableByteChannel that can read the object data

Throws:

IOException

create

@Deprecated
public WritableByteChannel create(GcsPath path,
 String type)
                           throws IOException

Deprecated.

Use create(GcsPath, CreateOptions) instead.

Throws:

IOException

create

@Deprecated
public WritableByteChannel create(GcsPath path,
 String type,
 Integer uploadBufferSizeBytes)
                           throws IOException

Deprecated.

Use create(GcsPath, CreateOptions) instead.

Throws:

IOException

create

public WritableByteChannel create(GcsPath path,
 GcsUtil.CreateOptions options)
                           throws IOException

Creates an object in GCS and prepares for uploading its contents.

Parameters:

path - the GCS file to write to

options - to be used for creating and configuring file upload

Returns:

a WritableByteChannel that can be used to write data to the object.

Throws:

IOException

verifyBucketAccessible

public void verifyBucketAccessible(GcsPath path)
                            throws IOException

Checks whether the GCS bucket exists. Similar to bucketAccessible(GcsPath), but throws exception if the bucket is inaccessible due to permissions or does not exist.

Throws:

IOException

bucketAccessible

public boolean bucketAccessible(GcsPath path)
                         throws IOException

Returns whether the GCS bucket exists and is accessible.

Throws:

IOException

bucketOwner

public long bucketOwner(GcsPath path)
                 throws IOException

Returns the project number of the project which owns this bucket. If the bucket exists, it must be accessible otherwise the permissions exception will be propagated. If the bucket does not exist, an exception will be thrown.

Throws:

IOException

createBucket

public void createBucket(String projectId,
 Bucket bucket)
                  throws IOException

Creates a Bucket under the specified project in Cloud Storage or propagates an exception.

Throws:

IOException

getBucket

public @Nullable Bucket getBucket(GcsPath path)
                           throws IOException

Get the Bucket from Cloud Storage path or propagates an exception.

Throws:

IOException

removeBucket

public void removeBucket(Bucket bucket)
                  throws IOException

Remove an empty Bucket in Cloud Storage or propagates an exception.

Throws:

IOException

copy

public void copy(Iterable<String> srcFilenames,
 Iterable<String> destFilenames)
          throws IOException

Throws:

IOException

rename

public void rename(Iterable<String> srcFilenames,
 Iterable<String> destFilenames,
 MoveOptions... moveOptions)
            throws IOException

Throws:

IOException

remove

public void remove(Collection<String> filenames)
            throws IOException

Throws:

IOException