org.apache.beam.sdk.extensions.sql.meta.provider.kafka

Class BeamKafkaTable

java.lang.Object

org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable

org.apache.beam.sdk.extensions.sql.meta.provider.kafka.BeamKafkaTable

All Implemented Interfaces:

java.io.Serializable, BeamSqlTable

Direct Known Subclasses:

BeamKafkaCSVTable

public abstract class BeamKafkaTable
extends BaseBeamTable
implements java.io.Serializable

BeamKafkaTable represent a Kafka topic, as source or target. Need to extend to convert between BeamSqlRow and KV<byte[], byte[]>.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable

beamRecordSqlType

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BeamKafkaTable(BeamRecordSqlType beamSqlRowType)
	BeamKafkaTable(BeamRecordSqlType beamSqlRowType, java.util.List<org.apache.kafka.common.TopicPartition> topicPartitions, java.lang.String bootstrapServers)
	BeamKafkaTable(BeamRecordSqlType beamSqlRowType, java.lang.String bootstrapServers, java.util.List<java.lang.String> topics)

Method Summary

Modifier and Type	Method and Description
PCollection<BeamRecord>	buildIOReader(Pipeline pipeline) create a PCollection<BeamSqlRow> from source.
PTransform<? super PCollection<BeamRecord>,PDone>	buildIOWriter() create a IO.write() instance to write to target.
java.lang.String	getBootstrapServers()
abstract PTransform<PCollection<KV<byte[],byte[]>>,PCollection<BeamRecord>>	getPTransformForInput()
abstract PTransform<PCollection<BeamRecord>,PCollection<KV<byte[],byte[]>>>	getPTransformForOutput()
BeamIOType	getSourceType() In Beam SQL, there's no difference between a batch query and a streaming query.
java.util.List<java.lang.String>	getTopics()
BeamKafkaTable	updateConsumerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

Methods inherited from class org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable

getRowType

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BeamKafkaTable

protected BeamKafkaTable(BeamRecordSqlType beamSqlRowType)

BeamKafkaTable

public BeamKafkaTable(BeamRecordSqlType beamSqlRowType,
                      java.lang.String bootstrapServers,
                      java.util.List<java.lang.String> topics)

BeamKafkaTable

public BeamKafkaTable(BeamRecordSqlType beamSqlRowType,
                      java.util.List<org.apache.kafka.common.TopicPartition> topicPartitions,
                      java.lang.String bootstrapServers)

Method Detail

updateConsumerProperties

public BeamKafkaTable updateConsumerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

getSourceType

public BeamIOType getSourceType()

Description copied from interface: BeamSqlTable

In Beam SQL, there's no difference between a batch query and a streaming query. BeamIOType is used to validate the sources.

Specified by:

getSourceType in interface BeamSqlTable

getPTransformForInput

public abstract PTransform<PCollection<KV<byte[],byte[]>>,PCollection<BeamRecord>> getPTransformForInput()

getPTransformForOutput

public abstract PTransform<PCollection<BeamRecord>,PCollection<KV<byte[],byte[]>>> getPTransformForOutput()

buildIOReader

public PCollection<BeamRecord> buildIOReader(Pipeline pipeline)

Description copied from interface: BeamSqlTable

create a PCollection<BeamSqlRow> from source.

Specified by:

buildIOReader in interface BeamSqlTable

buildIOWriter

public PTransform<? super PCollection<BeamRecord>,PDone> buildIOWriter()

Description copied from interface: BeamSqlTable

create a IO.write() instance to write to target.

Specified by:

buildIOWriter in interface BeamSqlTable

getBootstrapServers

public java.lang.String getBootstrapServers()

getTopics

public java.util.List<java.lang.String> getTopics()