Package org.apache.beam.io.debezium

Class DebeziumIO

java.lang.Object

org.apache.beam.io.debezium.DebeziumIO

public class DebeziumIO
extends Object

Utility class which exposes an implementation read() and a Debezium configuration.

Quick Overview

This class lets Beam users connect to their existing Debezium implementations in an easy way.

Any Kafka connector supported by Debezium should work fine with this IO.

The following connectors were tested and worked well in some simple scenarios:

• MySQL
• PostgreSQL
• SQLServer
• DB2

Usage example

Support is currently experimental. One of the known issues is that the connector does not preserve the offset on a worker crash or restart, causing it to retrieve all the data from the beginning again. See Issue #28248 for details.

Connect to a Debezium - MySQL database and run a Pipeline

     private static final ConnectorConfiguration mySqlConnectorConfig = ConnectorConfiguration
             .create()
             .withUsername("uname")
             .withPassword("pwd123")
             .withHostName("127.0.0.1")
             .withPort("3306")
             .withConnectorClass(MySqlConnector.class)
             .withConnectionProperty("database.server.id", "184054")
             .withConnectionProperty("database.server.name", "serverid")
             .withConnectionProperty("database.history", DebeziumSDFDatabaseHistory.class.getName())
             .withConnectionProperty("include.schema.changes", "false");

      PipelineOptions options = PipelineOptionsFactory.create();
      Pipeline p = Pipeline.create(options);
      p.apply(DebeziumIO.read()
               .withConnectorConfiguration(mySqlConnectorConfig)
               .withFormatFunction(new SourceRecordJson.SourceRecordJsonMapper())
       ).setCoder(StringUtf8Coder.of());
       p.run().waitUntilFinish();
 

In this example we are using KafkaSourceConsumerFn.DebeziumSDFDatabaseHistory to handle the Database history.

Dependencies

User may work with any of the supported Debezium Connectors above mentioned

See Debezium Connectors for more info.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DebeziumIO.ConnectorConfiguration	A POJO describing a Debezium configuration.
static class	DebeziumIO.Read<T>	Implementation of read().

Method Summary

Modifier and Type	Method	Description
static <T> DebeziumIO.Read<T>	read()	Read data from a Debezium source.
static DebeziumIO.Read<String>	readAsJson()	Read data from Debezium source and convert a Kafka SourceRecord into a JSON string using SourceRecordJson.SourceRecordJsonMapper as default function mapper.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

read

public static <T> DebeziumIO.Read<T> read()

Read data from a Debezium source.

Type Parameters:

T - Type of the data to be read.

readAsJson

public static DebeziumIO.Read<String> readAsJson()

Read data from Debezium source and convert a Kafka SourceRecord into a JSON string using SourceRecordJson.SourceRecordJsonMapper as default function mapper.

Returns:

Reader object of String.