org.apache.beam.sdk.io.elasticsearch

Class ElasticsearchIO

java.lang.Object

org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO

@Experimental(value=SOURCE_SINK)
public class ElasticsearchIO
extends java.lang.Object

Transforms for reading and writing data from/to Elasticsearch.

Reading from Elasticsearch

ElasticsearchIO.read() returns a bounded PCollection<String> representing JSON documents.

To configure the read(), you have to provide a connection configuration containing the HTTP address of the instances, an index name and a type. The following example illustrates options for configuring the source:

 pipeline.apply(ElasticsearchIO.read().withConnectionConfiguration(
    ElasticsearchIO.ConnectionConfiguration.create("http://host:9200", "my-index", "my-type")
 )

 

The connection configuration also accepts optional configuration: withUsername() and withPassword().

You can also specify a query on the read() using withQuery().

Writing to Elasticsearch

To write documents to Elasticsearch, use ElasticsearchIO.write(), which writes JSON documents from a PCollection<String> (which can be bounded or unbounded).

To configure ElasticsearchIO.write(), similar to the read, you have to provide a connection configuration. For instance:

 pipeline
   .apply(...)
   .apply(ElasticsearchIO.write().withConnectionConfiguration(
      ElasticsearchIO.ConnectionConfiguration.create("http://host:9200", "my-index", "my-type")
   )

 

Optionally, you can provide withBatchSize() and withBatchSizeBytes() to specify the size of the write batch in number of documents or in bytes.

Optionally, you can provide an ElasticsearchIO.Write.FieldValueExtractFn using withIdFn() that will be run to extract the id value out of the provided document rather than using the document id auto-generated by Elasticsearch.

Optionally, you can provide ElasticsearchIO.Write.FieldValueExtractFn using withIndexFn() or withTypeFn() to enable per-document routing to the target Elasticsearch index (all versions) and type (version > 6). Support for type routing was removed in Elasticsearch 6 (see https://www.elastic.co/blog/index-type-parent-child-join-now-future-in-elasticsearch)

When {withUsePartialUpdate()} is enabled, the input document must contain an id field and withIdFn() must be used to allow its extraction by the ElasticsearchIO.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ElasticsearchIO.BoundedElasticsearchSource A BoundedSource reading from Elasticsearch.
static class	ElasticsearchIO.ConnectionConfiguration A POJO describing a connection configuration to Elasticsearch.
static class	ElasticsearchIO.Read A PTransform reading data from Elasticsearch.
static class	ElasticsearchIO.RetryConfiguration A POJO encapsulating a configuration for retry behavior when issuing requests to ES.
static class	ElasticsearchIO.Write A PTransform writing data to Elasticsearch.

Method Summary

Modifier and Type	Method and Description
static ElasticsearchIO.Read	read()
static ElasticsearchIO.Write	write()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

read

public static ElasticsearchIO.Read read()

write

public static ElasticsearchIO.Write write()