@Experimental(value=SOURCE_SINK) public class ClickHouseIO extends java.lang.Object
To write to ClickHouse, use write(String, String)
, which writes elements
from input PCollection
. It's required that your ClickHouse cluster already has table you
are going to insert into.
pipeline
.apply(...)
.apply(
ClickHouseIO.<POJO>write("jdbc:clickhouse:localhost:8123/default", "my_table"));
Optionally, you can provide connection settings, for instance, specify insert block size with
ClickHouseIO.Write.withMaxInsertBlockSize(long)
, or configure number of retries with ClickHouseIO.Write.withMaxRetries(int)
.
ClickHouseIO.Write.withMaxInsertBlockSize(long)
.
Deduplication is performed using checksums of inserted blocks.
ClickHouse | Beam |
---|---|
TableSchema.TypeName.FLOAT32 | Schema.TypeName#FLOAT |
TableSchema.TypeName.FLOAT64 | Schema.TypeName#DOUBLE |
TableSchema.TypeName.FIXEDSTRING | FixedBytes |
TableSchema.TypeName.INT8 | Schema.TypeName#BYTE |
TableSchema.TypeName.INT16 | Schema.TypeName#INT16 |
TableSchema.TypeName.INT32 | Schema.TypeName#INT32 |
TableSchema.TypeName.INT64 | Schema.TypeName#INT64 |
TableSchema.TypeName.STRING | Schema.TypeName#STRING |
TableSchema.TypeName.UINT8 | Schema.TypeName#INT16 |
TableSchema.TypeName.UINT16 | Schema.TypeName#INT32 |
TableSchema.TypeName.UINT32 | Schema.TypeName#INT64 |
TableSchema.TypeName.UINT64 | Schema.TypeName#INT64 |
TableSchema.TypeName.DATE | Schema.TypeName#DATETIME |
TableSchema.TypeName.DATETIME | Schema.TypeName#DATETIME |
TableSchema.TypeName.ARRAY | Schema.TypeName#ARRAY |
TableSchema.TypeName.ENUM8 | Schema.TypeName#STRING |
TableSchema.TypeName.ENUM16 | Schema.TypeName#STRING |
Nested rows should be unnested using Select.flattenedSchema()
. Type casting should be
done using Cast
before ClickHouseIO
.
Modifier and Type | Class and Description |
---|---|
static class |
ClickHouseIO.Write<T>
A
PTransform to write to ClickHouse. |
Modifier and Type | Field and Description |
---|---|
static Duration |
DEFAULT_INITIAL_BACKOFF |
static Duration |
DEFAULT_MAX_CUMULATIVE_BACKOFF |
static long |
DEFAULT_MAX_INSERT_BLOCK_SIZE |
static int |
DEFAULT_MAX_RETRIES |
Constructor and Description |
---|
ClickHouseIO() |
Modifier and Type | Method and Description |
---|---|
static TableSchema |
getTableSchema(java.lang.String jdbcUrl,
java.lang.String table)
Returns
TableSchema for a given table. |
static <T> ClickHouseIO.Write<T> |
write(java.lang.String jdbcUrl,
java.lang.String table) |
public static final long DEFAULT_MAX_INSERT_BLOCK_SIZE
public static final int DEFAULT_MAX_RETRIES
public static final Duration DEFAULT_MAX_CUMULATIVE_BACKOFF
public static final Duration DEFAULT_INITIAL_BACKOFF
public static <T> ClickHouseIO.Write<T> write(java.lang.String jdbcUrl, java.lang.String table)
public static TableSchema getTableSchema(java.lang.String jdbcUrl, java.lang.String table)
TableSchema
for a given table.jdbcUrl
- jdbc connection urltable
- table name