org.apache.beam.sdk.io.kafka

Class KafkaIO.Write<K,V>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<KV<K,V>>,PDone>

org.apache.beam.sdk.io.kafka.KafkaIO.Write<K,V>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

KafkaIO

public abstract static class KafkaIO.Write<K,V>
extends PTransform<PCollection<KV<K,V>>,PDone>

A PTransform to write to a Kafka topic with KVs . See KafkaIO for more information on usage and configuration.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaIO.Write.External Exposes KafkaIO.Write as an external transform for cross-language usage.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.Class<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write>	AUTOVALUE_CLASS

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<KV<K,V>> input) Override this method to specify how this PTransform should be expanded on the given InputT.
abstract KafkaIO.WriteRecords<K,V>	getWriteRecordsTransform()
void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
KafkaIO.Write<K,V>	updateProducerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates) Deprecated. as of version 2.13. Use withProducerConfigUpdates(Map) instead.
void	validate(@Nullable PipelineOptions options) Called before running the Pipeline to verify this transform is fully and correctly specified.
PTransform<PCollection<V>,PDone>	values() Writes just the values to Kafka.
KafkaIO.Write<K,V>	withBadRecordErrorHandler(ErrorHandler<BadRecord,?> badRecordErrorHandler) Configure a ErrorHandler.BadRecordErrorHandler for sending records to if they fail to serialize when being sent to Kafka.
KafkaIO.Write<K,V>	withBootstrapServers(java.lang.String bootstrapServers) Wrapper method over KafkaIO.WriteRecords.withBootstrapServers(String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withConsumerFactoryFn(SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,? extends Consumer<?,?>> consumerFactoryFn) Wrapper method over KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withEOS(int numShards, java.lang.String sinkGroupId) Wrapper method over KafkaIO.WriteRecords.withEOS(int, String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withGCPApplicationDefaultCredentials() Creates and sets the Application Default Credentials for a Kafka producer.
KafkaIO.Write<K,V>	withInputTimestamp() Wrapper method over KafkaIO.WriteRecords.withInputTimestamp(), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withKeySerializer(java.lang.Class<? extends Serializer<K>> keySerializer) Wrapper method over KafkaIO.WriteRecords.withKeySerializer(Class), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withProducerConfigUpdates(java.util.Map<java.lang.String,java.lang.Object> configUpdates) Update configuration for the producer.
KafkaIO.Write<K,V>	withProducerFactoryFn(SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,Producer<K,V>> producerFactoryFn) Wrapper method over KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withPublishTimestampFunction(KafkaPublishTimestampFunction<KV<K,V>> timestampFunction) Deprecated. use KafkaIO.WriteRecords and ProducerRecords to set publish timestamp.
KafkaIO.Write<K,V>	withTopic(java.lang.String topic) Wrapper method over KafkaIO.WriteRecords.withTopic(String), used to keep the compatibility with old API based on KV type of element.
KafkaIO.Write<K,V>	withValueSerializer(java.lang.Class<? extends Serializer<V>> valueSerializer) Wrapper method over KafkaIO.WriteRecords.withValueSerializer(Class), used to keep the compatibility with old API based on KV type of element.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

AUTOVALUE_CLASS

public static final java.lang.Class<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write> AUTOVALUE_CLASS

Constructor Detail

Write

public Write()

Method Detail

getWriteRecordsTransform

public abstract KafkaIO.WriteRecords<K,V> getWriteRecordsTransform()

withBootstrapServers

public KafkaIO.Write<K,V> withBootstrapServers(java.lang.String bootstrapServers)

Wrapper method over KafkaIO.WriteRecords.withBootstrapServers(String), used to keep the compatibility with old API based on KV type of element.

withTopic

public KafkaIO.Write<K,V> withTopic(java.lang.String topic)

Wrapper method over KafkaIO.WriteRecords.withTopic(String), used to keep the compatibility with old API based on KV type of element.

withKeySerializer

public KafkaIO.Write<K,V> withKeySerializer(java.lang.Class<? extends Serializer<K>> keySerializer)

Wrapper method over KafkaIO.WriteRecords.withKeySerializer(Class), used to keep the compatibility with old API based on KV type of element.

withValueSerializer

public KafkaIO.Write<K,V> withValueSerializer(java.lang.Class<? extends Serializer<V>> valueSerializer)

Wrapper method over KafkaIO.WriteRecords.withValueSerializer(Class), used to keep the compatibility with old API based on KV type of element.

withProducerFactoryFn

public KafkaIO.Write<K,V> withProducerFactoryFn(SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,Producer<K,V>> producerFactoryFn)

Wrapper method over KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.

withInputTimestamp

public KafkaIO.Write<K,V> withInputTimestamp()

Wrapper method over KafkaIO.WriteRecords.withInputTimestamp(), used to keep the compatibility with old API based on KV type of element.

withPublishTimestampFunction

@Deprecated
public KafkaIO.Write<K,V> withPublishTimestampFunction(KafkaPublishTimestampFunction<KV<K,V>> timestampFunction)

Deprecated. use KafkaIO.WriteRecords and ProducerRecords to set publish timestamp.

Wrapper method over KafkaIO.WriteRecords.withPublishTimestampFunction(KafkaPublishTimestampFunction), used to keep the compatibility with old API based on KV type of element.

withEOS

public KafkaIO.Write<K,V> withEOS(int numShards,
                                  java.lang.String sinkGroupId)

Wrapper method over KafkaIO.WriteRecords.withEOS(int, String), used to keep the compatibility with old API based on KV type of element.

withConsumerFactoryFn

public KafkaIO.Write<K,V> withConsumerFactoryFn(SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,? extends Consumer<?,?>> consumerFactoryFn)

Wrapper method over KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.

updateProducerProperties

@Deprecated
public KafkaIO.Write<K,V> updateProducerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

Deprecated. as of version 2.13. Use withProducerConfigUpdates(Map) instead.

Adds the given producer properties, overriding old values of properties with the same key.

withProducerConfigUpdates

public KafkaIO.Write<K,V> withProducerConfigUpdates(java.util.Map<java.lang.String,java.lang.Object> configUpdates)

Update configuration for the producer. Note that the default producer properties will not be completely overridden. This method only updates the value which has the same key.

By default, the producer uses the configuration from KafkaIO.WriteRecords.DEFAULT_PRODUCER_PROPERTIES.

withBadRecordErrorHandler

public KafkaIO.Write<K,V> withBadRecordErrorHandler(ErrorHandler<BadRecord,?> badRecordErrorHandler)

Configure a ErrorHandler.BadRecordErrorHandler for sending records to if they fail to serialize when being sent to Kafka.

withGCPApplicationDefaultCredentials

public KafkaIO.Write<K,V> withGCPApplicationDefaultCredentials()

Creates and sets the Application Default Credentials for a Kafka producer. This allows the consumer to be authenticated with a Google Kafka Server using OAuth.

expand

public PDone expand(PCollection<KV<K,V>> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<KV<K,V>>,PDone>

validate

public void validate(@Nullable PipelineOptions options)

Description copied from class: PTransform

Called before running the Pipeline to verify this transform is fully and correctly specified.

By default, does nothing.

Overrides:

validate in class PTransform<PCollection<KV<K,V>>,PDone>

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Description copied from class: PTransform

Register display data for the given transform or component.

populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

By default, does not register any display data. Implementors may override this method to provide their own display data.

Specified by:

populateDisplayData in interface HasDisplayData

Overrides:

populateDisplayData in class PTransform<PCollection<KV<K,V>>,PDone>

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData

values

public PTransform<PCollection<V>,PDone> values()

Writes just the values to Kafka. This is useful for writing collections of values rather thank KVs.