public abstract static class KafkaIO.Write<K,V> extends PTransform<PCollection<KV<K,V>>,PDone>
PTransform
to write to a Kafka topic with KVs . See KafkaIO
for more
information on usage and configuration.Modifier and Type | Class and Description |
---|---|
static class |
KafkaIO.Write.External
Exposes
KafkaIO.Write as an external transform for cross-language usage. |
Modifier and Type | Field and Description |
---|---|
static java.lang.Class<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write> |
AUTOVALUE_CLASS |
annotations, displayData, name, resourceHints
Constructor and Description |
---|
Write() |
Modifier and Type | Method and Description |
---|---|
PDone |
expand(PCollection<KV<K,V>> input)
Override this method to specify how this
PTransform should be expanded on the given
InputT . |
abstract KafkaIO.WriteRecords<K,V> |
getWriteRecordsTransform() |
void |
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.
|
KafkaIO.Write<K,V> |
updateProducerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)
Deprecated.
as of version 2.13. Use
withProducerConfigUpdates(Map) instead. |
void |
validate(@Nullable PipelineOptions options)
Called before running the Pipeline to verify this transform is fully and correctly specified.
|
PTransform<PCollection<V>,PDone> |
values()
Writes just the values to Kafka.
|
KafkaIO.Write<K,V> |
withBadRecordErrorHandler(ErrorHandler<BadRecord,?> badRecordErrorHandler)
Configure a
ErrorHandler.BadRecordErrorHandler for sending records to if they fail to serialize
when being sent to Kafka. |
KafkaIO.Write<K,V> |
withBootstrapServers(java.lang.String bootstrapServers)
Wrapper method over
KafkaIO.WriteRecords.withBootstrapServers(String) , used to keep the
compatibility with old API based on KV type of element. |
KafkaIO.Write<K,V> |
withConsumerFactoryFn(SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,? extends Consumer<?,?>> consumerFactoryFn)
Wrapper method over
KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction) , used to
keep the compatibility with old API based on KV type of element. |
KafkaIO.Write<K,V> |
withEOS(int numShards,
java.lang.String sinkGroupId)
Wrapper method over
KafkaIO.WriteRecords.withEOS(int, String) , used to keep the compatibility
with old API based on KV type of element. |
KafkaIO.Write<K,V> |
withInputTimestamp()
Wrapper method over
KafkaIO.WriteRecords.withInputTimestamp() , used to keep the compatibility
with old API based on KV type of element. |
KafkaIO.Write<K,V> |
withKeySerializer(java.lang.Class<? extends Serializer<K>> keySerializer)
Wrapper method over
KafkaIO.WriteRecords.withKeySerializer(Class) , used to keep the
compatibility with old API based on KV type of element. |
KafkaIO.Write<K,V> |
withProducerConfigUpdates(java.util.Map<java.lang.String,java.lang.Object> configUpdates)
Update configuration for the producer.
|
KafkaIO.Write<K,V> |
withProducerFactoryFn(SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,Producer<K,V>> producerFactoryFn)
Wrapper method over
KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction) , used to
keep the compatibility with old API based on KV type of element. |
KafkaIO.Write<K,V> |
withPublishTimestampFunction(KafkaPublishTimestampFunction<KV<K,V>> timestampFunction)
Deprecated.
use
KafkaIO.WriteRecords and ProducerRecords to set publish timestamp. |
KafkaIO.Write<K,V> |
withTopic(java.lang.String topic)
Wrapper method over
KafkaIO.WriteRecords.withTopic(String) , used to keep the compatibility
with old API based on KV type of element. |
KafkaIO.Write<K,V> |
withValueSerializer(java.lang.Class<? extends Serializer<V>> valueSerializer)
Wrapper method over
KafkaIO.WriteRecords.withValueSerializer(Class) , used to keep the
compatibility with old API based on KV type of element. |
addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setDisplayData, setResourceHints, toString, validate
public static final java.lang.Class<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write> AUTOVALUE_CLASS
public abstract KafkaIO.WriteRecords<K,V> getWriteRecordsTransform()
public KafkaIO.Write<K,V> withBootstrapServers(java.lang.String bootstrapServers)
KafkaIO.WriteRecords.withBootstrapServers(String)
, used to keep the
compatibility with old API based on KV type of element.public KafkaIO.Write<K,V> withTopic(java.lang.String topic)
KafkaIO.WriteRecords.withTopic(String)
, used to keep the compatibility
with old API based on KV type of element.public KafkaIO.Write<K,V> withKeySerializer(java.lang.Class<? extends Serializer<K>> keySerializer)
KafkaIO.WriteRecords.withKeySerializer(Class)
, used to keep the
compatibility with old API based on KV type of element.public KafkaIO.Write<K,V> withValueSerializer(java.lang.Class<? extends Serializer<V>> valueSerializer)
KafkaIO.WriteRecords.withValueSerializer(Class)
, used to keep the
compatibility with old API based on KV type of element.public KafkaIO.Write<K,V> withProducerFactoryFn(SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,Producer<K,V>> producerFactoryFn)
KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction)
, used to
keep the compatibility with old API based on KV type of element.public KafkaIO.Write<K,V> withInputTimestamp()
KafkaIO.WriteRecords.withInputTimestamp()
, used to keep the compatibility
with old API based on KV type of element.@Deprecated public KafkaIO.Write<K,V> withPublishTimestampFunction(KafkaPublishTimestampFunction<KV<K,V>> timestampFunction)
KafkaIO.WriteRecords
and ProducerRecords
to set publish timestamp.KafkaIO.WriteRecords.withPublishTimestampFunction(KafkaPublishTimestampFunction)
, used to keep the
compatibility with old API based on KV type of element.public KafkaIO.Write<K,V> withEOS(int numShards, java.lang.String sinkGroupId)
KafkaIO.WriteRecords.withEOS(int, String)
, used to keep the compatibility
with old API based on KV type of element.public KafkaIO.Write<K,V> withConsumerFactoryFn(SerializableFunction<java.util.Map<java.lang.String,java.lang.Object>,? extends Consumer<?,?>> consumerFactoryFn)
KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction)
, used to
keep the compatibility with old API based on KV type of element.@Deprecated public KafkaIO.Write<K,V> updateProducerProperties(java.util.Map<java.lang.String,java.lang.Object> configUpdates)
withProducerConfigUpdates(Map)
instead.public KafkaIO.Write<K,V> withProducerConfigUpdates(java.util.Map<java.lang.String,java.lang.Object> configUpdates)
By default, the producer uses the configuration from KafkaIO.WriteRecords.DEFAULT_PRODUCER_PROPERTIES
.
public KafkaIO.Write<K,V> withBadRecordErrorHandler(ErrorHandler<BadRecord,?> badRecordErrorHandler)
ErrorHandler.BadRecordErrorHandler
for sending records to if they fail to serialize
when being sent to Kafka.public PDone expand(PCollection<KV<K,V>> input)
PTransform
PTransform
should be expanded on the given
InputT
.
NOTE: This method should not be called directly. Instead apply the PTransform
should
be applied to the InputT
using the apply
method.
Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
expand
in class PTransform<PCollection<KV<K,V>>,PDone>
public void validate(@Nullable PipelineOptions options)
PTransform
By default, does nothing.
validate
in class PTransform<PCollection<KV<K,V>>,PDone>
public void populateDisplayData(DisplayData.Builder builder)
PTransform
populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect
display data via DisplayData.from(HasDisplayData)
. Implementations may call super.populateDisplayData(builder)
in order to register display data in the current namespace,
but should otherwise use subcomponent.populateDisplayData(builder)
to use the namespace
of the subcomponent.
By default, does not register any display data. Implementors may override this method to provide their own display data.
populateDisplayData
in interface HasDisplayData
populateDisplayData
in class PTransform<PCollection<KV<K,V>>,PDone>
builder
- The builder to populate with display data.HasDisplayData
public PTransform<PCollection<V>,PDone> values()
KV
s.