Package org.apache.beam.sdk.io.kafka

Class KafkaReadSchemaTransformConfiguration

java.lang.Object

org.apache.beam.sdk.io.kafka.KafkaReadSchemaTransformConfiguration

@DefaultSchema(AutoValueSchema.class)
public abstract class KafkaReadSchemaTransformConfiguration
extends Object

Configuration for reading from a Kafka topic.

Internal only: This class is actively being worked on, and it will likely change. We provide no backwards compatibility guarantees, and it should not be implemented outside the Beam repository.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	KafkaReadSchemaTransformConfiguration.Builder	Builder for the KafkaReadSchemaTransformConfiguration.

Field Summary

Fields

Modifier and Type	Field	Description
static final Set<String>	VALID_DATA_FORMATS
static final String	VALID_FORMATS_STR
static final Set<String>	VALID_START_OFFSET_VALUES

Constructor Summary

Constructors

Constructor	Description
KafkaReadSchemaTransformConfiguration()

Method Summary

Modifier and Type	Method	Description
static KafkaReadSchemaTransformConfiguration.Builder	builder()	Instantiates a KafkaReadSchemaTransformConfiguration.Builder instance.
abstract Boolean	getAllowDuplicates()
abstract String	getAutoOffsetResetConfig()
abstract String	getBootstrapServers()	Sets the bootstrap servers for the Kafka consumer.
abstract String	getConfluentSchemaRegistrySubject()
abstract String	getConfluentSchemaRegistryUrl()
abstract Map<String,String>	getConsumerConfigUpdates()
abstract ErrorHandling	getErrorHandling()
abstract String	getFileDescriptorPath()
abstract String	getFormat()
abstract Integer	getMaxReadTimeSeconds()
abstract String	getMessageName()
abstract Boolean	getOffsetDeduplication()
abstract Boolean	getRedistributeByRecordKey()
abstract Boolean	getRedistributed()
abstract Integer	getRedistributeNumKeys()
abstract String	getSchema()
abstract String	getTopic()	Sets the topic from which to read.
void	validate()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

VALID_START_OFFSET_VALUES

public static final Set<String> VALID_START_OFFSET_VALUES

VALID_FORMATS_STR

public static final String VALID_FORMATS_STR

See Also:

• Constant Field Values

VALID_DATA_FORMATS

public static final Set<String> VALID_DATA_FORMATS

Constructor Details

KafkaReadSchemaTransformConfiguration

public KafkaReadSchemaTransformConfiguration()

Method Details

validate

public void validate()

builder

public static KafkaReadSchemaTransformConfiguration.Builder builder()

Instantiates a KafkaReadSchemaTransformConfiguration.Builder instance.

getBootstrapServers

@SchemaFieldDescription("A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping\u2014this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form `host1:port1,host2:port2,...`")
@SchemaFieldNumber("0")
public abstract String getBootstrapServers()

Sets the bootstrap servers for the Kafka consumer.

getConfluentSchemaRegistryUrl

@SchemaFieldNumber("1")
@Nullable
public abstract String getConfluentSchemaRegistryUrl()

getFormat

@SchemaFieldDescription("The encoding format for the data stored in Kafka. Valid options are: RAW,STRING,AVRO,JSON,PROTO")
@SchemaFieldNumber("2")
@Nullable
public abstract String getFormat()

getConfluentSchemaRegistrySubject

@SchemaFieldNumber("3")
@Nullable
public abstract String getConfluentSchemaRegistrySubject()

getSchema

@SchemaFieldDescription("The schema in which the data is encoded in the Kafka topic. For AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/). If a URL to Confluent Schema Registry is provided, then this field is ignored, and the schema is fetched from Confluent Schema Registry.")
@SchemaFieldNumber("4")
@Nullable
public abstract String getSchema()

getFileDescriptorPath

@SchemaFieldDescription("The path to the Protocol Buffer File Descriptor Set file. This file is used for schema definition and message serialization.")
@SchemaFieldNumber("5")
@Nullable
public abstract String getFileDescriptorPath()

getMessageName

@SchemaFieldDescription("The name of the Protocol Buffer message to be used for schema extraction and data conversion.")
@SchemaFieldNumber("6")
@Nullable
public abstract String getMessageName()

getAutoOffsetResetConfig

@SchemaFieldDescription("What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server. (1) earliest: automatically reset the offset to the earliest offset. (2) latest: automatically reset the offset to the latest offset (3) none: throw exception to the consumer if no previous offset is found for the consumer\u2019s group")
@SchemaFieldNumber("7")
@Nullable
public abstract String getAutoOffsetResetConfig()

getConsumerConfigUpdates

@SchemaFieldDescription("A list of key-value pairs that act as configuration parameters for Kafka consumers. Most of these configurations will not be needed, but if you need to customize your Kafka consumer, you may use this. See a detailed list: https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html")
@SchemaFieldNumber("8")
@Nullable
public abstract Map<String,String> getConsumerConfigUpdates()

getTopic

@SchemaFieldNumber("9")
public abstract String getTopic()

Sets the topic from which to read.

getMaxReadTimeSeconds

@SchemaFieldDescription("Upper bound of how long to read from Kafka.")
@SchemaFieldNumber("10")
@Nullable
public abstract Integer getMaxReadTimeSeconds()

getErrorHandling

@SchemaFieldDescription("This option specifies whether and where to output unwritable rows.")
@SchemaFieldNumber("11")
@Nullable
public abstract ErrorHandling getErrorHandling()

getRedistributed

@SchemaFieldDescription("If the Kafka read should be redistributed.")
@SchemaFieldNumber("12")
@Nullable
public abstract Boolean getRedistributed()

getAllowDuplicates

@SchemaFieldDescription("If the Kafka read allows duplicates.")
@SchemaFieldNumber("13")
@Nullable
public abstract Boolean getAllowDuplicates()

getRedistributeNumKeys

@SchemaFieldDescription("The number of keys for redistributing Kafka inputs.")
@SchemaFieldNumber("14")
@Nullable
public abstract Integer getRedistributeNumKeys()

getOffsetDeduplication

@SchemaFieldDescription("If the redistribute is using offset deduplication mode.")
@SchemaFieldNumber("15")
@Nullable
public abstract Boolean getOffsetDeduplication()

getRedistributeByRecordKey

@SchemaFieldDescription("If the redistribute keys by the Kafka record key.")
@SchemaFieldNumber("16")
@Nullable
public abstract Boolean getRedistributeByRecordKey()