Package org.apache.beam.sdk.extensions.sql.meta.provider.kafka

Class BeamKafkaTable

java.lang.Object

org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable

org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable

org.apache.beam.sdk.extensions.sql.meta.provider.kafka.BeamKafkaTable

All Implemented Interfaces:

Serializable, BeamSqlTable

Direct Known Subclasses:

BeamKafkaCSVTable, PayloadSerializerKafkaTable

public abstract class BeamKafkaTable
extends SchemaBaseBeamTable

BeamKafkaTable represent a Kafka topic, as source or target. Need to extend to convert between BeamSqlRow and KV<byte[], byte[]>.

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected int	numberOfRecordsForRate

Fields inherited from class org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable

schema

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BeamKafkaTable(Schema beamSchema)
	BeamKafkaTable(Schema beamSchema, String bootstrapServers, List<String> topics)
	BeamKafkaTable(Schema beamSchema, String bootstrapServers, List<String> topics, TimestampPolicyFactory timestampPolicyFactory)
	BeamKafkaTable(Schema beamSchema, List<TopicPartition> topicPartitions, String bootstrapServers)
	BeamKafkaTable(Schema beamSchema, List<TopicPartition> topicPartitions, String bootstrapServers, TimestampPolicyFactory timestampPolicyFactory)

Method Summary

Modifier and Type	Method	Description
PCollection<Row>	buildIOReader(PBegin begin)	create a PCollection<Row> from source.
POutput	buildIOWriter(PCollection<Row> input)	create a IO.write() instance to write to target.
protected KafkaIO.Read<byte[],byte[]>	createKafkaRead()
String	getBootstrapServers()
Map<String,Object>	getConfigUpdates()
protected abstract PTransform<PCollection<KafkaRecord<byte[],byte[]>>,PCollection<Row>>	getPTransformForInput()
protected abstract PTransform<PCollection<Row>,PCollection<ProducerRecord<byte[],byte[]>>>	getPTransformForOutput()
BeamTableStatistics	getTableStatistics(PipelineOptions options)	Estimates the number of rows or the rate for unbounded Tables.
TimestampPolicyFactory	getTimestampPolicyFactory()
List<String>	getTopics()
PCollection.IsBounded	isBounded()	Whether this table is bounded (known to be finite) or unbounded (may or may not be finite).
BeamKafkaTable	updateConsumerProperties(Map<String,Object> configUpdates)

Methods inherited from class org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable

getSchema

Methods inherited from class org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable

buildIOReader, constructFilter, supportsProjects

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

numberOfRecordsForRate

protected int numberOfRecordsForRate

Constructor Details

BeamKafkaTable

protected BeamKafkaTable(Schema beamSchema)

BeamKafkaTable

public BeamKafkaTable(Schema beamSchema,
 String bootstrapServers,
 List<String> topics)

BeamKafkaTable

public BeamKafkaTable(Schema beamSchema,
 String bootstrapServers,
 List<String> topics,
 TimestampPolicyFactory timestampPolicyFactory)

BeamKafkaTable

public BeamKafkaTable(Schema beamSchema,
 List<TopicPartition> topicPartitions,
 String bootstrapServers)

BeamKafkaTable

public BeamKafkaTable(Schema beamSchema,
 List<TopicPartition> topicPartitions,
 String bootstrapServers,
 TimestampPolicyFactory timestampPolicyFactory)

Method Details

updateConsumerProperties

public BeamKafkaTable updateConsumerProperties(Map<String,Object> configUpdates)

isBounded

public PCollection.IsBounded isBounded()

Description copied from interface: BeamSqlTable

Whether this table is bounded (known to be finite) or unbounded (may or may not be finite).

getPTransformForInput

protected abstract PTransform<PCollection<KafkaRecord<byte[],byte[]>>,PCollection<Row>> getPTransformForInput()

getPTransformForOutput

protected abstract PTransform<PCollection<Row>,PCollection<ProducerRecord<byte[],byte[]>>> getPTransformForOutput()

buildIOReader

public PCollection<Row> buildIOReader(PBegin begin)

Description copied from interface: BeamSqlTable

create a PCollection<Row> from source.

createKafkaRead

protected KafkaIO.Read<byte[],byte[]> createKafkaRead()

buildIOWriter

public POutput buildIOWriter(PCollection<Row> input)

Description copied from interface: BeamSqlTable

create a IO.write() instance to write to target.

getBootstrapServers

public String getBootstrapServers()

getTopics

public List<String> getTopics()

getConfigUpdates

public Map<String,Object> getConfigUpdates()

getTimestampPolicyFactory

public TimestampPolicyFactory getTimestampPolicyFactory()

getTableStatistics

public BeamTableStatistics getTableStatistics(PipelineOptions options)

Description copied from interface: BeamSqlTable

Estimates the number of rows or the rate for unbounded Tables. If it is not possible to estimate the row count or rate it will return BeamTableStatistics.BOUNDED_UNKNOWN.

Specified by:

getTableStatistics in interface BeamSqlTable

Overrides:

getTableStatistics in class BaseBeamTable