org.apache.beam.sdk.extensions.sql.meta.provider.kafka

Class BeamKafkaTable

java.lang.Object

org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable

org.apache.beam.sdk.extensions.sql.meta.provider.kafka.BeamKafkaTable

All Implemented Interfaces:

java.io.Serializable, BeamSqlTable

Direct Known Subclasses:

BeamKafkaCSVTable

public abstract class BeamKafkaTable
extends BaseBeamTable

BeamKafkaTable represent a Kafka topic, as source or target. Need to extend to convert between BeamSqlRow and KV<byte[], byte[]>.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable

schema

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BeamKafkaTable(Schema beamSchema)
	BeamKafkaTable(Schema beamSchema, java.util.List<org.apache.kafka.common.TopicPartition> topicPartitions, java.lang.String bootstrapServers)
	BeamKafkaTable(Schema beamSchema, java.lang.String bootstrapServers, java.util.List<java.lang.String> topics)

Method Summary

Modifier and Type	Method and Description
PCollection<Row>	buildIOReader(Pipeline pipeline) create a PCollection<BeamSqlRow> from source.
PTransform<? super PCollection<Row>,POutput>	buildIOWriter() create a IO.write() instance to write to target.
java.lang.String	getBootstrapServers()
abstract PTransform<PCollection<KV<byte[],byte[]>>,PCollection<Row>>	getPTransformForInput()
abstract PTransform<PCollection<Row>,PCollection<KV<byte[],byte[]>>>	getPTransformForOutput()
BeamIOType	getSourceType() In Beam SQL, there's no difference between a batch query and a streaming query.
java.util.List<java.lang.String>	getTopics()
BeamKafkaTable	updateConsumerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

Methods inherited from class org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable

getSchema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BeamKafkaTable

protected BeamKafkaTable(Schema beamSchema)

BeamKafkaTable

public BeamKafkaTable(Schema beamSchema,
                      java.lang.String bootstrapServers,
                      java.util.List<java.lang.String> topics)

BeamKafkaTable

public BeamKafkaTable(Schema beamSchema,
                      java.util.List<org.apache.kafka.common.TopicPartition> topicPartitions,
                      java.lang.String bootstrapServers)

Method Detail

updateConsumerProperties

public BeamKafkaTable updateConsumerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

getSourceType

public BeamIOType getSourceType()

Description copied from interface: BeamSqlTable

In Beam SQL, there's no difference between a batch query and a streaming query. BeamIOType is used to validate the sources.

getPTransformForInput

public abstract PTransform<PCollection<KV<byte[],byte[]>>,PCollection<Row>> getPTransformForInput()

getPTransformForOutput

public abstract PTransform<PCollection<Row>,PCollection<KV<byte[],byte[]>>> getPTransformForOutput()

buildIOReader

public PCollection<Row> buildIOReader(Pipeline pipeline)

Description copied from interface: BeamSqlTable

create a PCollection<BeamSqlRow> from source.

buildIOWriter

public PTransform<? super PCollection<Row>,POutput> buildIOWriter()

Description copied from interface: BeamSqlTable

create a IO.write() instance to write to target.

getBootstrapServers

public java.lang.String getBootstrapServers()

getTopics

public java.util.List<java.lang.String> getTopics()