Class GcsUtil
java.lang.Object
org.apache.beam.sdk.extensions.gcp.util.GcsUtil
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic classstatic classDeprecated.static classstatic class -
Field Summary
Fields -
Method Summary
Modifier and TypeMethodDescriptionbooleanbucketAccessible(GcsPath path) longbucketOwner(GcsPath path) voidvoidcopy(Iterable<GcsPath> srcPaths, Iterable<GcsPath> dstPaths, org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.OverwriteStrategy strategy) experimental api.voidexperimental api.Deprecated.Usecreate(GcsPath, CreateOptions)instead.Deprecated.Usecreate(GcsPath, CreateOptions)instead.create(GcsPath path, GcsUtil.CreateOptions options) voidcreateBucket(com.google.cloud.storage.BucketInfo bucketInfo, com.google.cloud.storage.Storage.BucketTargetOption... options) voidcreateBucket(String projectId, Bucket bucket) Deprecated.createV2(GcsPath path, GcsUtil.CreateOptions options, com.google.cloud.storage.Storage.BlobWriteOption... writeOptions) longcom.google.cloud.storage.BlobList<org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.BlobResult> Deprecated.@Nullable com.google.cloud.storage.BucketgetBucketWithOptions(GcsPath path, com.google.cloud.storage.Storage.BucketGetOption... options) static StringgetNonWildcardPrefix(String globExp) Deprecated.useGcsPath.getNonWildcardPrefix(String)instead.Deprecated.getObjects(List<GcsPath> gcsPaths) Deprecated.static booleanisWildcard(GcsPath spec) Deprecated.useGcsPath.isWildcard(GcsPath)instead.com.google.api.gax.paging.Page<com.google.cloud.storage.Blob> listBlobs(String bucket, String prefix, @Nullable String pageToken, com.google.cloud.storage.Storage.BlobListOption... options) com.google.api.gax.paging.Page<com.google.cloud.storage.Blob> listBlobs(String bucket, String prefix, @Nullable String pageToken, @Nullable String delimiter, com.google.cloud.storage.Storage.BlobListOption... options) listObjects(String bucket, String prefix, @Nullable String pageToken) Deprecated.Deprecated.voidremove(Iterable<GcsPath> paths, org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.MissingStrategy strategy) experimental api.voidremove(Collection<String> filenames) voidremoveBucket(Bucket bucket) Deprecated.voidremoveBucket(com.google.cloud.storage.BucketInfo bucketInfo) voidexperimental api.voidvoidrename(Iterable<GcsPath> srcPaths, Iterable<GcsPath> dstPaths, org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.MissingStrategy srcMissing, org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.OverwriteStrategy dstOverwrite) experimental api.voidexperimental api.void
-
Field Details
-
METRIC_NAMESPACE
Namespace for every GCS metric. The namespace is dropped when Dataflow exports counters to Cloud Monitoring, so the layer is carried by the metric name instead:gcs_http_*for transport-level counters andgcs_op_*for operation-level ones.- See Also:
-
DISABLE_GCSUTIL_V2_EXPERIMENT
Experiment that falls back to GcsUtil V1 (gcsio). V2 (java-storage) is the default, and this experiment is the escape hatch should V2 misbehave for a pipeline.- See Also:
-
USE_GCSUTIL_V2_EXPERIMENT
Experiment that used to opt into GcsUtil V2. V2 is now the default, so this is a no-op kept so that existing pipelines that still pass it keep working.DISABLE_GCSUTIL_V2_EXPERIMENTtakes precedence if both are set.- See Also:
-
USE_GRPC_FOR_GCS_EXPERIMENT
Experiment that makes GcsUtil V1 talk to GCS over gRPC. Only V1 (gcsio) supports it, so it also falls back to V1, the same asDISABLE_GCSUTIL_V2_EXPERIMENT.- See Also:
-
-
Method Details
-
getNonWildcardPrefix
Deprecated.useGcsPath.getNonWildcardPrefix(String)instead. -
isWildcard
Deprecated.useGcsPath.isWildcard(GcsPath)instead. -
expand
- Throws:
IOException
-
fileSize
- Throws:
IOException
-
getObject
Deprecated.- Throws:
IOException
-
getBlob
public com.google.cloud.storage.Blob getBlob(GcsPath gcsPath, com.google.cloud.storage.Storage.BlobGetOption... options) throws IOException - Throws:
IOException
-
getObjects
@Deprecated public List<GcsUtil.StorageObjectOrIOException> getObjects(List<GcsPath> gcsPaths) throws IOException Deprecated.- Throws:
IOException
-
getBlobs
public List<org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.BlobResult> getBlobs(Iterable<GcsPath> gcsPaths, com.google.cloud.storage.Storage.BlobGetOption... options) throws IOException - Throws:
IOException
-
listObjects
@Deprecated public Objects listObjects(String bucket, String prefix, @Nullable String pageToken) throws IOException Deprecated.- Throws:
IOException
-
listObjects
@Deprecated public Objects listObjects(String bucket, String prefix, @Nullable String pageToken, @Nullable String delimiter) throws IOException Deprecated.- Throws:
IOException
-
listBlobs
public com.google.api.gax.paging.Page<com.google.cloud.storage.Blob> listBlobs(String bucket, String prefix, @Nullable String pageToken, com.google.cloud.storage.Storage.BlobListOption... options) throws IOException - Throws:
IOException
-
listBlobs
public com.google.api.gax.paging.Page<com.google.cloud.storage.Blob> listBlobs(String bucket, String prefix, @Nullable String pageToken, @Nullable String delimiter, com.google.cloud.storage.Storage.BlobListOption... options) throws IOException - Throws:
IOException
-
open
- Throws:
IOException
-
openV2
public SeekableByteChannel openV2(GcsPath path, com.google.cloud.storage.Storage.BlobSourceOption... options) throws IOException - Throws:
IOException
-
create
Deprecated.Usecreate(GcsPath, CreateOptions)instead.- Throws:
IOException
-
create
@Deprecated public WritableByteChannel create(GcsPath path, String type, Integer uploadBufferSizeBytes) throws IOException Deprecated.Usecreate(GcsPath, CreateOptions)instead.- Throws:
IOException
-
create
- Throws:
IOException
-
createV2
public WritableByteChannel createV2(GcsPath path, GcsUtil.CreateOptions options, com.google.cloud.storage.Storage.BlobWriteOption... writeOptions) throws IOException - Throws:
IOException
-
verifyBucketAccessible
- Throws:
IOException
-
bucketAccessible
- Throws:
IOException
-
bucketOwner
- Throws:
IOException
-
createBucket
Deprecated.- Throws:
IOException
-
createBucket
public void createBucket(com.google.cloud.storage.BucketInfo bucketInfo, com.google.cloud.storage.Storage.BucketTargetOption... options) throws IOException - Throws:
IOException
-
getBucket
Deprecated.- Throws:
IOException
-
getBucketWithOptions
public @Nullable com.google.cloud.storage.Bucket getBucketWithOptions(GcsPath path, com.google.cloud.storage.Storage.BucketGetOption... options) throws IOException - Throws:
IOException
-
removeBucket
Deprecated.- Throws:
IOException
-
removeBucket
- Throws:
IOException
-
copy
- Throws:
IOException
-
copyV2
experimental api.- Throws:
IOException
-
copy
public void copy(Iterable<GcsPath> srcPaths, Iterable<GcsPath> dstPaths, org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.OverwriteStrategy strategy) throws IOException experimental api.- Throws:
IOException
-
rename
public void rename(Iterable<String> srcFilenames, Iterable<String> destFilenames, MoveOptions... moveOptions) throws IOException - Throws:
IOException
-
renameV2
public void renameV2(Iterable<GcsPath> srcPaths, Iterable<GcsPath> dstPaths, MoveOptions... moveOptions) throws IOException experimental api.- Throws:
IOException
-
rename
public void rename(Iterable<GcsPath> srcPaths, Iterable<GcsPath> dstPaths, org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.MissingStrategy srcMissing, org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.OverwriteStrategy dstOverwrite) throws IOException experimental api.- Throws:
IOException
-
remove
- Throws:
IOException
-
removeV2
experimental api.- Throws:
IOException
-
remove
public void remove(Iterable<GcsPath> paths, org.apache.beam.sdk.extensions.gcp.util.GcsUtilV2.MissingStrategy strategy) throws IOException experimental api.- Throws:
IOException
-
GcsUtilAPI accepts this type, so an instance cannot be used for anything.