Package org.apache.beam.sdk.io.clickhouse

Class ClickHouseIO

java.lang.Object
org.apache.beam.sdk.io.clickhouse.ClickHouseIO
public class ClickHouseIO extends Object
An IO to write to ClickHouse.

Writing to ClickHouse

To write to ClickHouse, use write(String, String, String), which writes elements from input PCollection. It's required that your ClickHouse cluster already has table you are going to insert into. 


 // New way (recommended):
 Properties props = new Properties();
 props.setProperty("user", "admin");
 props.setProperty("password", "secret");

 pipeline
   .apply(...)
   .apply(
     ClickHouseIO.<POJO>write("http://localhost:8123", "default", "my_table")
       .withProperties(props));

 // Old way (deprecated):
 pipeline
   .apply(...)
   .apply(
     ClickHouseIO.<POJO>write("jdbc:clickhouse://localhost:8123/default", "my_table"));

Optionally, you can provide connection settings, for instance, specify insert block size with ClickHouseIO.Write.withMaxInsertBlockSize(long), or configure number of retries with ClickHouseIO.Write.withMaxRetries(int).

Deduplication

Deduplication is performed by ClickHouse if inserting to ReplicatedMergeTree or Distributed table on top of ReplicatedMergeTree. Without replication, inserting into regular MergeTree can produce duplicates, if insert fails, and then successfully retries. However, each block is inserted atomically, and you can configure block size with ClickHouseIO.Write.withMaxInsertBlockSize(long).

Deduplication is performed using checksums of inserted blocks. For SharedMergeTree tables in ClickHouse Cloud, deduplication behavior is similar to ReplicatedMergeTree. For more information about deduplication, please visit the Deduplication strategies documentation

Mapping between Beam and ClickHouse types

ClickHouse Beam
TableSchema.TypeName.FLOAT32 Schema.TypeName.FLOAT
TableSchema.TypeName.FLOAT64 Schema.TypeName.DOUBLE
TableSchema.TypeName.FIXEDSTRING FixedBytes
TableSchema.TypeName.INT8 Schema.TypeName.BYTE
TableSchema.TypeName.INT16 Schema.TypeName.INT16
TableSchema.TypeName.INT32 Schema.TypeName.INT32
TableSchema.TypeName.INT64 Schema.TypeName.INT64
TableSchema.TypeName.STRING Schema.TypeName.STRING
TableSchema.TypeName.UINT8 Schema.TypeName.INT16
TableSchema.TypeName.UINT16 Schema.TypeName.INT32
TableSchema.TypeName.UINT32 Schema.TypeName.INT64
TableSchema.TypeName.UINT64 Schema.TypeName.INT64
TableSchema.TypeName.DATE Schema.TypeName.DATETIME
TableSchema.TypeName.DATETIME Schema.TypeName.DATETIME
TableSchema.TypeName.ARRAY Schema.TypeName.ARRAY
TableSchema.TypeName.ENUM8 Schema.TypeName.STRING
TableSchema.TypeName.ENUM16 Schema.TypeName.STRING
TableSchema.TypeName.BOOL Schema.TypeName.BOOLEAN
TableSchema.TypeName.TUPLE Schema.TypeName.ROW

Nullable row columns are supported through Nullable type in ClickHouse. Low cardinality hint is supported through LowCardinality DataType in ClickHouse.

Nested rows should be unnested using Select.flattenedSchema(). Type casting should be done using Cast before ClickHouseIO.

  • Field Details

    • DEFAULT_MAX_INSERT_BLOCK_SIZE

      public static final long DEFAULT_MAX_INSERT_BLOCK_SIZE
      See Also:

    • DEFAULT_MAX_RETRIES

      public static final int DEFAULT_MAX_RETRIES
      See Also:

    • DEFAULT_MAX_CUMULATIVE_BACKOFF

      public static final Duration DEFAULT_MAX_CUMULATIVE_BACKOFF

    • DEFAULT_INITIAL_BACKOFF

      public static final Duration DEFAULT_INITIAL_BACKOFF

  • Constructor Details

    • ClickHouseIO

      public ClickHouseIO()

  • Method Details

    • write

      @Deprecated public static <T> ClickHouseIO.Write<T> write(String jdbcUrl, String table)
      Deprecated.
      Use write(String, String, String) with explicit URL, database, and table
      Creates a write transform using a JDBC URL format.

      Deprecated: Use write(String, String, String) instead with separate URL, database, and table parameters.

      This method is provided for backward compatibility. It parses the JDBC URL to extract the connection URL, database name, and any connection properties specified in the query string. Properties can be overridden later using ClickHouseIO.Write.withProperties(Properties).

      Example: 

      
 // Old way (deprecated):
 ClickHouseIO.write("jdbc:clickhouse://localhost:8123/mydb?user=admin&password=secret", "table")

 // New way:
 ClickHouseIO.write("http://localhost:8123", "mydb", "table")
   .withProperties(props)

      Property Precedence: Properties from the JDBC URL can be overridden by calling ClickHouseIO.Write.withProperties(Properties). Later calls to withProperties() override earlier settings.

      Parameters:
      jdbcUrl - JDBC connection URL (e.g., jdbc:clickhouse://host:port/database?param=value)
      table - table name
      Returns:
      a PTransform writing data to ClickHouse

    • write

      public static <T> ClickHouseIO.Write<T> write(String clickHouseUrl, String database, String table)

    • getTableSchema

      @Deprecated public static TableSchema getTableSchema(String jdbcUrl, String table)
      Deprecated.
      Use getTableSchema(String, String, String, Properties) with explicit parameters
      Returns TableSchema for a given table using JDBC URL format.

      Deprecated: Use getTableSchema(String, String, String, Properties) instead with separate URL, database, table, and properties parameters.

      This method parses the JDBC URL to extract connection details and properties. For new code, use the explicit parameter version for better clarity and control.

      Example migration: 

      
 // Old way (deprecated):
 TableSchema schema = ClickHouseIO.getTableSchema(
     "jdbc:clickhouse://localhost:8123/mydb?user=admin", "my_table");

 // New way:
 Properties props = new Properties();
 props.setProperty("user", "admin");
 TableSchema schema = ClickHouseIO.getTableSchema(
     "http://localhost:8123", "mydb", "my_table", props);
      Parameters:
      jdbcUrl - JDBC connection URL (e.g., jdbc:clickhouse://host:port/database?param=value)
      table - table name
      Returns:
      table schema

    • getTableSchema

      public static TableSchema getTableSchema(String clickHouseUrl, String database, String table, Properties properties)
      Returns TableSchema for a given table using ClickHouse Java Client.
      Parameters:
      clickHouseUrl - ClickHouse connection url
      database - ClickHouse database
      table - table name
      properties - connection properties
      Returns:
      table schema
      Since:
      2.72.0