org.apache.beam.io.debezium

Class DebeziumIO

java.lang.Object

org.apache.beam.io.debezium.DebeziumIO

@Experimental(value=SOURCE_SINK)
public class DebeziumIO
extends java.lang.Object

Utility class which exposes an implementation read() and a Debezium configuration.

Quick Overview

This class lets Beam users connect to their existing Debezium implementations in an easy way.

Any Kafka connector supported by Debezium should work fine with this IO.

The following connectors were tested and worked well in some simple scenarios:

• MySQL
• PostgreSQL
• SQLServer
• DB2

Usage example

Connect to a Debezium - MySQL database and run a Pipeline

     private static final ConnectorConfiguration mySqlConnectorConfig = ConnectorConfiguration
             .create()
             .withUsername("uname")
             .withPassword("pwd123")
             .withHostName("127.0.0.1")
             .withPort("3306")
             .withConnectorClass(MySqlConnector.class)
             .withConnectionProperty("database.server.id", "184054")
             .withConnectionProperty("database.server.name", "serverid")
             .withConnectionProperty("database.include.list", "dbname")
             .withConnectionProperty("database.history", DebeziumSDFDatabaseHistory.class.getName())
             .withConnectionProperty("include.schema.changes", "false");

      PipelineOptions options = PipelineOptionsFactory.create();
      Pipeline p = Pipeline.create(options);
      p.apply(DebeziumIO.read()
               .withConnectorConfiguration(mySqlConnectorConfig)
               .withFormatFunction(new SourceRecordJson.SourceRecordJsonMapper())
       ).setCoder(StringUtf8Coder.of());
       p.run().waitUntilFinish();
 

In this example we are using KafkaSourceConsumerFn.DebeziumSDFDatabaseHistory to handle the Database history.

Dependencies

User may work with any of the supported Debezium Connectors above mentioned

See Debezium Connectors for more info.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DebeziumIO.ConnectorConfiguration A POJO describing a Debezium configuration.
static class	DebeziumIO.Read<T> Implementation of read().

Method Summary

Modifier and Type	Method and Description
static <T> DebeziumIO.Read<T>	read() Read data from a Debezium source.
static DebeziumIO.Read<java.lang.String>	readAsJson() Read data from Debezium source and convert a Kafka SourceRecord into a JSON string using SourceRecordJson.SourceRecordJsonMapper as default function mapper.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

read

public static <T> DebeziumIO.Read<T> read()

Read data from a Debezium source.

Type Parameters:

T - Type of the data to be read.

readAsJson

public static DebeziumIO.Read<java.lang.String> readAsJson()

Read data from Debezium source and convert a Kafka SourceRecord into a JSON string using SourceRecordJson.SourceRecordJsonMapper as default function mapper.

Returns:

Reader object of String.