org.apache.beam.sdk.io.gcp.bigtable.changestreams.dao

Class MetadataTableAdminDao

java.lang.Object

org.apache.beam.sdk.io.gcp.bigtable.changestreams.dao.MetadataTableAdminDao

@Internal
public class MetadataTableAdminDao
extends java.lang.Object

Data access object for creating and dropping the metadata table.

The metadata table will be used to keep the state of the entire Beam pipeline as well as splitting and merging partitions.

Each Beam pipeline will create its own metadata table.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CF_CONTINUATION_TOKEN
static java.lang.String	CF_INITIAL_TOKEN
static java.lang.String	CF_LOCK
static java.lang.String	CF_MISSING_PARTITIONS
static java.lang.String	CF_PARENT_LOW_WATERMARKS
static java.lang.String	CF_PARENT_PARTITIONS
static java.lang.String	CF_SHOULD_DELETE
static java.lang.String	CF_VERSION
static java.lang.String	CF_WATERMARK
static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList<java.lang.String>	COLUMN_FAMILIES
static int	CURRENT_METADATA_TABLE_VERSION
static java.lang.String	DEFAULT_METADATA_TABLE_NAME
static com.google.protobuf.ByteString	DETECT_NEW_PARTITION_SUFFIX
static com.google.protobuf.ByteString	NEW_PARTITION_PREFIX
static java.lang.String	QUALIFIER_DEFAULT
static com.google.protobuf.ByteString	STREAM_PARTITION_PREFIX

Constructor Summary

Constructors

Constructor and Description
MetadataTableAdminDao(com.google.cloud.bigtable.admin.v2.BigtableTableAdminClient tableAdminClient, com.google.cloud.bigtable.admin.v2.BigtableInstanceAdminClient instanceAdminClient, java.lang.String changeStreamName, java.lang.String tableId)

Method Summary

Modifier and Type	Method and Description
void	cleanUpPrefix() Delete all the metadata rows starting with the change stream name prefix, except for detect new partition row because it signals the existence of a pipeline with the change stream name.
boolean	createMetadataTable() Create the metadata table if it does not exist yet.
boolean	doesMetadataTableExist()
com.google.protobuf.ByteString	getChangeStreamNamePrefix() Return the prefix used to identify the rows belonging to this job.
java.lang.String	getTableId() Return the metadata table name.
boolean	isAppProfileSingleClusterAndTransactional(java.lang.String appProfileId) Verify the app profile is for single cluster routing with allow single-row transactions enabled.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_METADATA_TABLE_NAME

public static final java.lang.String DEFAULT_METADATA_TABLE_NAME

See Also:

Constant Field Values

CF_INITIAL_TOKEN

public static final java.lang.String CF_INITIAL_TOKEN

See Also:

Constant Field Values

CF_PARENT_PARTITIONS

public static final java.lang.String CF_PARENT_PARTITIONS

See Also:

Constant Field Values

CF_PARENT_LOW_WATERMARKS

public static final java.lang.String CF_PARENT_LOW_WATERMARKS

See Also:

Constant Field Values

CF_WATERMARK

public static final java.lang.String CF_WATERMARK

See Also:

Constant Field Values

CF_CONTINUATION_TOKEN

public static final java.lang.String CF_CONTINUATION_TOKEN

See Also:

Constant Field Values

CF_LOCK

public static final java.lang.String CF_LOCK

See Also:

Constant Field Values

CF_MISSING_PARTITIONS

public static final java.lang.String CF_MISSING_PARTITIONS

See Also:

Constant Field Values

CF_VERSION

public static final java.lang.String CF_VERSION

See Also:

Constant Field Values

CF_SHOULD_DELETE

public static final java.lang.String CF_SHOULD_DELETE

See Also:

Constant Field Values

QUALIFIER_DEFAULT

public static final java.lang.String QUALIFIER_DEFAULT

See Also:

Constant Field Values

COLUMN_FAMILIES

public static final org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList<java.lang.String> COLUMN_FAMILIES

NEW_PARTITION_PREFIX

public static final com.google.protobuf.ByteString NEW_PARTITION_PREFIX

STREAM_PARTITION_PREFIX

public static final com.google.protobuf.ByteString STREAM_PARTITION_PREFIX

DETECT_NEW_PARTITION_SUFFIX

public static final com.google.protobuf.ByteString DETECT_NEW_PARTITION_SUFFIX

CURRENT_METADATA_TABLE_VERSION

public static final int CURRENT_METADATA_TABLE_VERSION

See Also:

Constant Field Values

Constructor Detail

MetadataTableAdminDao

public MetadataTableAdminDao(com.google.cloud.bigtable.admin.v2.BigtableTableAdminClient tableAdminClient,
                             com.google.cloud.bigtable.admin.v2.BigtableInstanceAdminClient instanceAdminClient,
                             java.lang.String changeStreamName,
                             java.lang.String tableId)

Method Detail

getChangeStreamNamePrefix

public com.google.protobuf.ByteString getChangeStreamNamePrefix()

Return the prefix used to identify the rows belonging to this job.

Returns:

the prefix used to identify the rows belonging to this job

getTableId

public java.lang.String getTableId()

Return the metadata table name.

Returns:

the metadata table name

isAppProfileSingleClusterAndTransactional

public boolean isAppProfileSingleClusterAndTransactional(java.lang.String appProfileId)

Verify the app profile is for single cluster routing with allow single-row transactions enabled. For metadata data operations, the app profile needs to be single cluster routing because it requires read-after-write consistency. Also, the operations depend on single row transactions operations like CheckAndMutateRow.

Returns:

true if the app profile is single-cluster and allows single-row transactions, otherwise false

doesMetadataTableExist

public boolean doesMetadataTableExist()

Returns:

true if metadata table exists, otherwise false.

createMetadataTable

public boolean createMetadataTable()

Create the metadata table if it does not exist yet. If the table does exist, verify all the column families exists, if not add those column families. This table only need to be created once per instance. All change streams jobs will use this table. This table is created in the same instance as the table being streamed. While we don't restrict access to the table, manually editing the table can lead to inconsistent beam jobs.

Returns:

true if the table was successfully created, otherwise false.

cleanUpPrefix

public void cleanUpPrefix()

Delete all the metadata rows starting with the change stream name prefix, except for detect new partition row because it signals the existence of a pipeline with the change stream name.