Package org.apache.beam.sdk.io.gcp.testing

Class FakeDatasetService

java.lang.Object

org.apache.beam.sdk.io.gcp.testing.FakeDatasetService

All Implemented Interfaces:

Serializable, AutoCloseable, BigQueryServices.DatasetService, BigQueryServices.WriteStreamService

@Internal
public class FakeDatasetService
extends Object
implements BigQueryServices.DatasetService, BigQueryServices.WriteStreamService, Serializable

A fake dataset service that can be serialized, for use in testReadFromTable.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.DatasetService

BigQueryServices.DatasetService.TableMetadataView

Constructor Summary

Constructors

Constructor	Description
FakeDatasetService()

Method Summary

Modifier and Type	Method	Description
void	close()
com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse>	commitWriteStreams(String tableUrn, Iterable<String> writeStreamNames)	Commit write streams of type PENDING.
void	createDataset(String projectId, String datasetId, String location, String description, Long defaultTableExpirationMs)	Create a Dataset with the given location, description and default expiration time for tables in the dataset (if null, tables don't expire).
void	createTable(Table table)	Creates the specified table if it does not exist.
com.google.cloud.bigquery.storage.v1.WriteStream	createWriteStream(String tableUrn, com.google.cloud.bigquery.storage.v1.WriteStream.Type type)	Create a Write Stream for use with the Storage Write API.
void	deleteDataset(String projectId, String datasetId)	Deletes the dataset specified by the datasetId value.
void	deleteTable(TableReference tableRef)	Deletes the table specified by tableId from the dataset.
void	failOnInsert(Map<TableRow,List<TableDataInsertAllResponse.InsertErrors>> insertErrors)	Cause a given TableRow object to fail when it's inserted.
com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse>	finalizeWriteStream(String streamName)	Finalize a write stream.
com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse>	flush(String streamName, long offset)	Flush a given stream up to the given offset.
List<String>	getAllIds(String projectId, String datasetId, String tableId)
List<TableRow>	getAllRows(String projectId, String datasetId, String tableId)
Dataset	getDataset(String projectId, String datasetId)	Gets the specified Dataset resource by dataset ID.
int	getInsertCount()
BigQueryServices.StreamAppendClient	getStreamAppendClient(String streamName, DescriptorProtos.DescriptorProto descriptor, boolean useConnectionPool, com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation missingValueInterpretation)	Create an append client for a given Storage API write stream.
Table	getTable(TableReference tableRef)	Gets the specified Table resource by table ID.
Table	getTable(TableReference tableRef, List<String> selectedFields)
Table	getTable(TableReference tableRef, List<String> selectedFields, BigQueryServices.DatasetService.TableMetadataView view)
Table	getTableImpl(TableReference tableRef, List<String> selectedFields, BigQueryServices.DatasetService.TableMetadataView view)
com.google.cloud.bigquery.storage.v1.TableSchema	getWriteStreamSchema(String streamName)
long	insertAll(TableReference ref, List<TableRow> rowList, List<String> insertIdList)
<T> long	insertAll(TableReference ref, List<FailsafeValueInSingleWindow<TableRow,TableRow>> rowList, List<String> insertIdList, InsertRetryPolicy retryPolicy, List<ValueInSingleWindow<T>> failedInserts, ErrorContainer<T> errorContainer, boolean skipInvalidRows, boolean ignoreUnknownValues, boolean ignoreInsertIds, List<ValueInSingleWindow<TableRow>> successfulRows)	Inserts TableRows with the specified insertIds if not null.
boolean	isTableEmpty(TableReference tableRef)	Returns true if the table is empty.
Table	patchTableDescription(TableReference tableReference, String tableDescription)	Patch BigQuery Table description.
void	setPrimaryKey(TableReference tableReference, List<String> columns)
void	setShouldFailRow(Function<TableRow,Boolean> shouldFailRow)
static void	setUp()
void	updateTableSchema(TableReference tableReference, TableSchema tableSchema)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

FakeDatasetService

public FakeDatasetService()

Method Details

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

setUp

public static void setUp()

setShouldFailRow

public void setShouldFailRow(Function<TableRow,Boolean> shouldFailRow)

getTable

public Table getTable(TableReference tableRef)
               throws InterruptedException,
IOException

Description copied from interface: BigQueryServices.DatasetService

Gets the specified Table resource by table ID.

Returns null if the table is not found.

Specified by:

getTable in interface BigQueryServices.DatasetService

Throws:

InterruptedException

IOException

getTable

public Table getTable(TableReference tableRef,
 @Nullable
 List<String> selectedFields)
               throws InterruptedException,
IOException

Specified by:

getTable in interface BigQueryServices.DatasetService

Throws:

InterruptedException

IOException

getTable

public Table getTable(TableReference tableRef,
 @Nullable
 List<String> selectedFields,
 @Nullable
 BigQueryServices.DatasetService.TableMetadataView view)
               throws InterruptedException,
IOException

Specified by:

getTable in interface BigQueryServices.DatasetService

Throws:

InterruptedException

IOException

getTableImpl

public Table getTableImpl(TableReference tableRef,
 @Nullable
 List<String> selectedFields,
 @Nullable
 BigQueryServices.DatasetService.TableMetadataView view)
                   throws InterruptedException,
IOException

Throws:

InterruptedException

IOException

getAllRows

public List<TableRow> getAllRows(String projectId,
 String datasetId,
 String tableId)
                          throws InterruptedException,
IOException

Throws:

InterruptedException

IOException

getAllIds

public List<String> getAllIds(String projectId,
 String datasetId,
 String tableId)
                       throws InterruptedException,
IOException

Throws:

InterruptedException

IOException

deleteTable

public void deleteTable(TableReference tableRef)
                 throws IOException,
InterruptedException

Description copied from interface: BigQueryServices.DatasetService

Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted.

Specified by:

deleteTable in interface BigQueryServices.DatasetService

Throws:

IOException

InterruptedException

createTable

public void createTable(Table table)
                 throws IOException

Description copied from interface: BigQueryServices.DatasetService

Creates the specified table if it does not exist.

Specified by:

createTable in interface BigQueryServices.DatasetService

Throws:

IOException

updateTableSchema

public void updateTableSchema(TableReference tableReference,
 TableSchema tableSchema)
                       throws IOException

Throws:

IOException

setPrimaryKey

public void setPrimaryKey(TableReference tableReference,
 List<String> columns)
                   throws IOException

Throws:

IOException

isTableEmpty

public boolean isTableEmpty(TableReference tableRef)
                     throws IOException,
InterruptedException

Description copied from interface: BigQueryServices.DatasetService

Returns true if the table is empty.

Specified by:

isTableEmpty in interface BigQueryServices.DatasetService

Throws:

IOException - if the table is not found.

InterruptedException

getDataset

public Dataset getDataset(String projectId,
 String datasetId)
                   throws IOException,
InterruptedException

Description copied from interface: BigQueryServices.DatasetService

Gets the specified Dataset resource by dataset ID.

Specified by:

getDataset in interface BigQueryServices.DatasetService

Throws:

IOException

InterruptedException

createDataset

public void createDataset(String projectId,
 String datasetId,
 String location,
 String description,
 Long defaultTableExpirationMs)
                   throws IOException,
InterruptedException

Description copied from interface: BigQueryServices.DatasetService

Create a Dataset with the given location, description and default expiration time for tables in the dataset (if null, tables don't expire).

Specified by:

createDataset in interface BigQueryServices.DatasetService

Throws:

IOException

InterruptedException

deleteDataset

public void deleteDataset(String projectId,
 String datasetId)
                   throws IOException,
InterruptedException

Description copied from interface: BigQueryServices.DatasetService

Deletes the dataset specified by the datasetId value.

Before you can delete a dataset, you must delete all its tables.

Specified by:

deleteDataset in interface BigQueryServices.DatasetService

Throws:

IOException

InterruptedException

getInsertCount

public int getInsertCount()

insertAll

public long insertAll(TableReference ref,
 List<TableRow> rowList,
 @Nullable
 List<String> insertIdList)
               throws IOException,
InterruptedException

Throws:

IOException

InterruptedException

insertAll

public <T> long insertAll(TableReference ref,
 List<FailsafeValueInSingleWindow<TableRow,TableRow>> rowList,
 @Nullable
 List<String> insertIdList,
 InsertRetryPolicy retryPolicy,
 List<ValueInSingleWindow<T>> failedInserts,
 ErrorContainer<T> errorContainer,
 boolean skipInvalidRows,
 boolean ignoreUnknownValues,
 boolean ignoreInsertIds,
 List<ValueInSingleWindow<TableRow>> successfulRows)
                   throws IOException,
InterruptedException

Description copied from interface: BigQueryServices.DatasetService

Inserts TableRows with the specified insertIds if not null.

If any insert fail permanently according to the retry policy, those rows are added to failedInserts.

Returns the total bytes count of TableRows.

Specified by:

insertAll in interface BigQueryServices.DatasetService

Throws:

IOException

InterruptedException

patchTableDescription

public Table patchTableDescription(TableReference tableReference,
 @Nullable
 String tableDescription)
                            throws IOException,
InterruptedException

Description copied from interface: BigQueryServices.DatasetService

Patch BigQuery Table description.

Specified by:

patchTableDescription in interface BigQueryServices.DatasetService

Throws:

IOException

InterruptedException

createWriteStream

public com.google.cloud.bigquery.storage.v1.WriteStream createWriteStream(String tableUrn,
 com.google.cloud.bigquery.storage.v1.WriteStream.Type type)
                                                                   throws InterruptedException

Description copied from interface: BigQueryServices.WriteStreamService

Create a Write Stream for use with the Storage Write API.

Specified by:

createWriteStream in interface BigQueryServices.WriteStreamService

Throws:

InterruptedException

getWriteStreamSchema

@Nullable
public com.google.cloud.bigquery.storage.v1.TableSchema getWriteStreamSchema(String streamName)

Specified by:

getWriteStreamSchema in interface BigQueryServices.WriteStreamService

getStreamAppendClient

public BigQueryServices.StreamAppendClient getStreamAppendClient(String streamName,
 DescriptorProtos.DescriptorProto descriptor,
 boolean useConnectionPool,
 com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation missingValueInterpretation)
                                                          throws Exception

Description copied from interface: BigQueryServices.WriteStreamService

Create an append client for a given Storage API write stream. The stream must be created first.

Specified by:

getStreamAppendClient in interface BigQueryServices.WriteStreamService

Throws:

Exception

flush

public com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse> flush(String streamName,
 long offset)

Description copied from interface: BigQueryServices.WriteStreamService

Flush a given stream up to the given offset. The stream must have type BUFFERED.

Specified by:

flush in interface BigQueryServices.WriteStreamService

finalizeWriteStream

public com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse> finalizeWriteStream(String streamName)

Description copied from interface: BigQueryServices.WriteStreamService

Finalize a write stream. After finalization, no more records can be appended to the stream.

Specified by:

finalizeWriteStream in interface BigQueryServices.WriteStreamService

commitWriteStreams

public com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse> commitWriteStreams(String tableUrn,
 Iterable<String> writeStreamNames)

Description copied from interface: BigQueryServices.WriteStreamService

Commit write streams of type PENDING. The streams must be finalized before committing.

Specified by:

commitWriteStreams in interface BigQueryServices.WriteStreamService

failOnInsert

public void failOnInsert(Map<TableRow,List<TableDataInsertAllResponse.InsertErrors>> insertErrors)

Cause a given TableRow object to fail when it's inserted. The errors link the list will be returned on subsequent retries, and the insert will succeed when the errors run out.