org.apache.beam.sdk.io.gcp.bigquery

Interface BigQueryServices.DatasetService

All Superinterfaces:

java.lang.AutoCloseable

All Known Implementing Classes:

BigQueryServicesImpl.DatasetServiceImpl, FakeDatasetService

Enclosing interface:

BigQueryServices

public static interface BigQueryServices.DatasetService
extends java.lang.AutoCloseable

An interface to get, create and delete Cloud BigQuery datasets and tables.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	BigQueryServices.DatasetService.TableMetadataView

Method Summary

Modifier and Type	Method and Description
com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse>	commitWriteStreams(java.lang.String tableUrn, java.lang.Iterable<java.lang.String> writeStreamNames) Commit write streams of type PENDING.
void	createDataset(java.lang.String projectId, java.lang.String datasetId, @Nullable java.lang.String location, @Nullable java.lang.String description, @Nullable java.lang.Long defaultTableExpirationMs) Create a Dataset with the given location, description and default expiration time for tables in the dataset (if null, tables don't expire).
void	createTable(Table table) Creates the specified table if it does not exist.
com.google.cloud.bigquery.storage.v1.WriteStream	createWriteStream(java.lang.String tableUrn, com.google.cloud.bigquery.storage.v1.WriteStream.Type type) Create a Write Stream for use with the Storage Write API.
void	deleteDataset(java.lang.String projectId, java.lang.String datasetId) Deletes the dataset specified by the datasetId value.
void	deleteTable(TableReference tableRef) Deletes the table specified by tableId from the dataset.
com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse>	finalizeWriteStream(java.lang.String streamName) Finalize a write stream.
com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse>	flush(java.lang.String streamName, long flushOffset) Flush a given stream up to the given offset.
Dataset	getDataset(java.lang.String projectId, java.lang.String datasetId) Gets the specified Dataset resource by dataset ID.
BigQueryServices.StreamAppendClient	getStreamAppendClient(java.lang.String streamName, com.google.protobuf.DescriptorProtos.DescriptorProto descriptor, boolean useConnectionPool, com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation missingValueInterpretation) Create an append client for a given Storage API write stream.
@Nullable Table	getTable(TableReference tableRef) Gets the specified Table resource by table ID.
@Nullable Table	getTable(TableReference tableRef, java.util.List<java.lang.String> selectedFields)
@Nullable Table	getTable(TableReference tableRef, java.util.List<java.lang.String> selectedFields, BigQueryServices.DatasetService.TableMetadataView view)
@Nullable com.google.cloud.bigquery.storage.v1.WriteStream	getWriteStream(java.lang.String writeStream)
<T> long	insertAll(TableReference ref, java.util.List<FailsafeValueInSingleWindow<TableRow,TableRow>> rowList, @Nullable java.util.List<java.lang.String> insertIdList, InsertRetryPolicy retryPolicy, java.util.List<ValueInSingleWindow<T>> failedInserts, ErrorContainer<T> errorContainer, boolean skipInvalidRows, boolean ignoreUnknownValues, boolean ignoreInsertIds, java.util.List<ValueInSingleWindow<TableRow>> successfulRows) Inserts TableRows with the specified insertIds if not null.
boolean	isTableEmpty(TableReference tableRef) Returns true if the table is empty.
Table	patchTableDescription(TableReference tableReference, @Nullable java.lang.String tableDescription) Patch BigQuery Table description.

Methods inherited from interface java.lang.AutoCloseable

close

Method Detail

getTable

@Nullable Table getTable(TableReference tableRef)
                  throws java.lang.InterruptedException,
                         java.io.IOException

Gets the specified Table resource by table ID.

Returns null if the table is not found.

Throws:

java.lang.InterruptedException

java.io.IOException

getTable

@Nullable Table getTable(TableReference tableRef,
                         java.util.List<java.lang.String> selectedFields)
                  throws java.lang.InterruptedException,
                         java.io.IOException

Throws:

java.lang.InterruptedException

java.io.IOException

getTable

@Nullable Table getTable(TableReference tableRef,
                         java.util.List<java.lang.String> selectedFields,
                         BigQueryServices.DatasetService.TableMetadataView view)
                  throws java.lang.InterruptedException,
                         java.io.IOException

Throws:

java.lang.InterruptedException

java.io.IOException

createTable

void createTable(Table table)
          throws java.lang.InterruptedException,
                 java.io.IOException

Creates the specified table if it does not exist.

Throws:

java.lang.InterruptedException

java.io.IOException

deleteTable

void deleteTable(TableReference tableRef)
          throws java.io.IOException,
                 java.lang.InterruptedException

Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted.

Throws:

java.io.IOException

java.lang.InterruptedException

isTableEmpty

boolean isTableEmpty(TableReference tableRef)
              throws java.io.IOException,
                     java.lang.InterruptedException

Returns true if the table is empty.

Throws:

java.io.IOException - if the table is not found.

java.lang.InterruptedException

getDataset

Dataset getDataset(java.lang.String projectId,
                   java.lang.String datasetId)
            throws java.io.IOException,
                   java.lang.InterruptedException

Gets the specified Dataset resource by dataset ID.

Throws:

java.io.IOException

java.lang.InterruptedException

createDataset

void createDataset(java.lang.String projectId,
                   java.lang.String datasetId,
                   @Nullable java.lang.String location,
                   @Nullable java.lang.String description,
                   @Nullable java.lang.Long defaultTableExpirationMs)
            throws java.io.IOException,
                   java.lang.InterruptedException

Create a Dataset with the given location, description and default expiration time for tables in the dataset (if null, tables don't expire).

Throws:

java.io.IOException

java.lang.InterruptedException

deleteDataset

void deleteDataset(java.lang.String projectId,
                   java.lang.String datasetId)
            throws java.io.IOException,
                   java.lang.InterruptedException

Deletes the dataset specified by the datasetId value.

Before you can delete a dataset, you must delete all its tables.

Throws:

java.io.IOException

java.lang.InterruptedException

insertAll

<T> long insertAll(TableReference ref,
                   java.util.List<FailsafeValueInSingleWindow<TableRow,TableRow>> rowList,
                   @Nullable java.util.List<java.lang.String> insertIdList,
                   InsertRetryPolicy retryPolicy,
                   java.util.List<ValueInSingleWindow<T>> failedInserts,
                   ErrorContainer<T> errorContainer,
                   boolean skipInvalidRows,
                   boolean ignoreUnknownValues,
                   boolean ignoreInsertIds,
                   java.util.List<ValueInSingleWindow<TableRow>> successfulRows)
            throws java.io.IOException,
                   java.lang.InterruptedException

Inserts TableRows with the specified insertIds if not null.

If any insert fail permanently according to the retry policy, those rows are added to failedInserts.

Returns the total bytes count of TableRows.

Throws:

java.io.IOException

java.lang.InterruptedException

patchTableDescription

Table patchTableDescription(TableReference tableReference,
                            @Nullable java.lang.String tableDescription)
                     throws java.io.IOException,
                            java.lang.InterruptedException

Patch BigQuery Table description.

Throws:

java.io.IOException

java.lang.InterruptedException

createWriteStream

com.google.cloud.bigquery.storage.v1.WriteStream createWriteStream(java.lang.String tableUrn,
                                                                   com.google.cloud.bigquery.storage.v1.WriteStream.Type type)
                                                            throws java.io.IOException,
                                                                   java.lang.InterruptedException

Create a Write Stream for use with the Storage Write API.

Throws:

java.io.IOException

java.lang.InterruptedException

getWriteStream

@Nullable com.google.cloud.bigquery.storage.v1.WriteStream getWriteStream(java.lang.String writeStream)

getStreamAppendClient

BigQueryServices.StreamAppendClient getStreamAppendClient(java.lang.String streamName,
                                                          com.google.protobuf.DescriptorProtos.DescriptorProto descriptor,
                                                          boolean useConnectionPool,
                                                          com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation missingValueInterpretation)
                                                   throws java.lang.Exception

Create an append client for a given Storage API write stream. The stream must be created first.

Throws:

java.lang.Exception

flush

com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse> flush(java.lang.String streamName,
                                                                                            long flushOffset)
                                                                                     throws java.io.IOException,
                                                                                            java.lang.InterruptedException

Flush a given stream up to the given offset. The stream must have type BUFFERED.

Throws:

java.io.IOException

java.lang.InterruptedException

finalizeWriteStream

com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse> finalizeWriteStream(java.lang.String streamName)

Finalize a write stream. After finalization, no more records can be appended to the stream.

commitWriteStreams

com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse> commitWriteStreams(java.lang.String tableUrn,
                                                                                                                       java.lang.Iterable<java.lang.String> writeStreamNames)

Commit write streams of type PENDING. The streams must be finalized before committing.