org.apache.beam.it.gcp.bigtable

Class BigtableResourceManager

java.lang.Object

org.apache.beam.it.gcp.bigtable.BigtableResourceManager

All Implemented Interfaces:

org.apache.beam.it.common.ResourceManager

public class BigtableResourceManager
extends java.lang.Object
implements org.apache.beam.it.common.ResourceManager

Client for managing Bigtable resources.

The class supports one instance, and multiple tables per manager object. An instance is created when the first table is created if one has not been created already.

The instance id is formed using testId. The instance id will be "{testId}-{ISO8601 time, microsecond precision}", with additional formatting. Note: If testId is more than 30 characters, a new testId will be formed for naming: {first 21 chars of long testId} + “-” + {8 char hash of testId}.

The class is thread-safe.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigtableResourceManager.Builder Builder for BigtableResourceManager.

Method Summary

Modifier and Type	Method and Description
static BigtableResourceManager.Builder	builder(java.lang.String testId, java.lang.String projectId, com.google.api.gax.core.CredentialsProvider credentialsProvider)
void	cleanupAll() Deletes all created resources (instance and tables) and cleans up all Bigtable clients, making the manager object unusable.
void	createAppProfile(java.lang.String appProfileId, boolean allowTransactionWrites, java.util.List<java.lang.String> clusters) Creates an application profile within the current instance
void	createInstance(java.util.List<BigtableResourceManagerCluster> clusters) Creates a Bigtable instance in which all clusters, nodes and tables will exist.
void	createTable(java.lang.String tableId, BigtableTableSpec bigtableTableSpec) Creates a table within the current instance given a table ID and a collection of column family names.
void	createTable(java.lang.String tableId, java.lang.Iterable<java.lang.String> columnFamilies) Creates a table within the current instance given a table ID and a collection of column family names.
void	createTable(java.lang.String tableId, java.lang.Iterable<java.lang.String> columnFamilies, org.threeten.bp.Duration maxAge) Creates a table within the current instance given a table ID and a collection of column family names.
java.util.List<java.lang.String>	getClusterNames() Get all the cluster names of the current instance.
java.lang.String	getInstanceId() Return the instance ID this Resource Manager uses to create and manage tables in.
java.lang.String	getProjectId() Returns the project ID this Resource Manager is configured to operate on.
com.google.common.collect.ImmutableList<com.google.cloud.bigtable.data.v2.models.Row>	readTable(java.lang.String tableId) Reads all the rows in a table.
com.google.common.collect.ImmutableList<com.google.cloud.bigtable.data.v2.models.Row>	readTable(java.lang.String tableId, java.lang.Long limit) Reads all the rows in a table.
void	write(java.lang.Iterable<com.google.cloud.bigtable.data.v2.models.RowMutation> tableRows) Writes a collection of table rows into one or more tables.
void	write(com.google.cloud.bigtable.data.v2.models.RowMutation tableRow) Writes a given row into a table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static BigtableResourceManager.Builder builder(java.lang.String testId,
                                                      java.lang.String projectId,
                                                      com.google.api.gax.core.CredentialsProvider credentialsProvider)

getProjectId

public java.lang.String getProjectId()

Returns the project ID this Resource Manager is configured to operate on.

Returns:

the project ID.

getInstanceId

public java.lang.String getInstanceId()

Return the instance ID this Resource Manager uses to create and manage tables in.

Returns:

the instance ID.

createInstance

public void createInstance(java.util.List<BigtableResourceManagerCluster> clusters)
                    throws BigtableResourceManagerException

Creates a Bigtable instance in which all clusters, nodes and tables will exist.

Parameters:

clusters - List of BigtableResourceManagerCluster objects to associate with the given Bigtable instance.

Throws:

BigtableResourceManagerException - if there is an error creating the instance in Bigtable.

createTable

public void createTable(java.lang.String tableId,
                        java.lang.Iterable<java.lang.String> columnFamilies)
                 throws BigtableResourceManagerException

Creates a table within the current instance given a table ID and a collection of column family names.

Bigtable has the capability to store multiple versions of data in a single cell, which are indexed using timestamp values. The timestamp can be set by Bigtable, with the default timestamp value being 1970-01-01, or can be set explicitly. The columns in the created table will be automatically garbage collected once they reach an age of 1-hour after the set timestamp.

Note: Implementations may do instance creation here, if one does not already exist.

Parameters:

tableId - The id of the table.

columnFamilies - A collection of column family names for the table.

Throws:

BigtableResourceManagerException - if there is an error creating the table in Bigtable.

createTable

public void createTable(java.lang.String tableId,
                        java.lang.Iterable<java.lang.String> columnFamilies,
                        org.threeten.bp.Duration maxAge)
                 throws BigtableResourceManagerException

Creates a table within the current instance given a table ID and a collection of column family names.

Bigtable has the capability to store multiple versions of data in a single cell, which are indexed using timestamp values. The timestamp can be set by Bigtable, with the default timestamp value being 1970-01-01, or can be set explicitly. The columns in the created table will be automatically garbage collected once they reach an age of maxAge after the set timestamp.

Note: Implementations may do instance creation here, if one does not already exist.

Parameters:

tableId - The id of the table.

columnFamilies - A collection of column family names for the table.

maxAge - Sets the maximum age the columns can persist before being garbage collected.

Throws:

BigtableResourceManagerException - if there is an error creating the table in Bigtable.

createTable

public void createTable(java.lang.String tableId,
                        BigtableTableSpec bigtableTableSpec)
                 throws BigtableResourceManagerException

Creates a table within the current instance given a table ID and a collection of column family names.

Bigtable has the capability to store multiple versions of data in a single cell, which are indexed using timestamp values. The timestamp can be set by Bigtable, with the default timestamp value being 1970-01-01, or can be set explicitly. The columns in the created table will be automatically garbage collected once they reach an age of maxAge after the set timestamp.

Note: Implementations may do instance creation here, if one does not already exist.

Parameters:

tableId - The id of the table.

bigtableTableSpec - Other table configurations

Throws:

BigtableResourceManagerException - if there is an error creating the table in Bigtable.

createAppProfile

public void createAppProfile(java.lang.String appProfileId,
                             boolean allowTransactionWrites,
                             java.util.List<java.lang.String> clusters)
                      throws BigtableResourceManagerException

Creates an application profile within the current instance

Note: Implementations may do instance creation here, if one does not already exist.

Parameters:

appProfileId - The id of the app profile.

allowTransactionWrites - Allows transactional writes when single cluster routing is enabled

clusters - Clusters where traffic is going to be routed. If more than one cluster is specified, a multi-cluster routing is used. A single-cluster routing is used when a single cluster is specified.

Throws:

BigtableResourceManagerException - if there is an error creating the application profile in Bigtable.

write

public void write(com.google.cloud.bigtable.data.v2.models.RowMutation tableRow)
           throws BigtableResourceManagerException

Writes a given row into a table. This method requires createTable(String, Iterable) to be called for the target table beforehand.

Parameters:

tableRow - A mutation object representing the table row.

Throws:

BigtableResourceManagerException - if method is called after resources have been cleaned up, if the manager object has no instance, if the table does not exist or if there is an IOException when attempting to retrieve the bigtable data client.

write

public void write(java.lang.Iterable<com.google.cloud.bigtable.data.v2.models.RowMutation> tableRows)
           throws BigtableResourceManagerException

Writes a collection of table rows into one or more tables. This method requires createTable(String, Iterable) to be called for the target table beforehand.

Parameters:

tableRows - A collection of mutation objects representing table rows.

Throws:

BigtableResourceManagerException - if method is called after resources have been cleaned up, if the manager object has no instance, if the table does not exist or if there is an IOException when attempting to retrieve the bigtable data client.

readTable

public com.google.common.collect.ImmutableList<com.google.cloud.bigtable.data.v2.models.Row> readTable(java.lang.String tableId)
                                                                                                throws BigtableResourceManagerException

Reads all the rows in a table. This method requires createTable(String, Iterable) to be called for the target table beforehand.

Parameters:

tableId - The id of table to read rows from.

Returns:

A List object containing all the rows in the table.

Throws:

BigtableResourceManagerException - if method is called after resources have been cleaned up, if the manager object has no instance, if the table does not exist or if there is an IOException when attempting to retrieve the bigtable data client.

readTable

public com.google.common.collect.ImmutableList<com.google.cloud.bigtable.data.v2.models.Row> readTable(java.lang.String tableId,
                                                                                                       @Nullable
                                                                                                       java.lang.Long limit)
                                                                                                throws BigtableResourceManagerException

Reads all the rows in a table. This method requires createTable(String, Iterable) to be called for the target table beforehand.

Parameters:

tableId - The id of table to read rows from.

limit - Limits the number of rows that can be returned

Returns:

A List object containing all the rows in the table.

Throws:

BigtableResourceManagerException - if method is called after resources have been cleaned up, if the manager object has no instance, if the table does not exist or if there is an IOException when attempting to retrieve the bigtable data client.

getClusterNames

public java.util.List<java.lang.String> getClusterNames()

Get all the cluster names of the current instance.

cleanupAll

public void cleanupAll()
                throws BigtableResourceManagerException

Deletes all created resources (instance and tables) and cleans up all Bigtable clients, making the manager object unusable.

If this Resource Manager was configured to use a static instance, the instance will not be cleaned up, but any created tables will be deleted.

Specified by:

cleanupAll in interface org.apache.beam.it.common.ResourceManager

Throws:

BigtableResourceManagerException - if there is an error deleting the instance or tables in Bigtable.