org.apache.beam.sdk.io.elasticsearch

Class ElasticsearchIO

java.lang.Object

org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO

@Experimental(value=SOURCE_SINK)
public class ElasticsearchIO
extends java.lang.Object

Transforms for reading and writing data from/to Elasticsearch. This IO is only compatible with Elasticsearch v2.x

Reading from Elasticsearch

ElasticsearchIO.read() returns a bounded PCollection<String> representing JSON documents.

To configure the read(), you have to provide a connection configuration containing the HTTP address of the instances, an index name and a type. The following example illustrates options for configuring the source:

 pipeline.apply(ElasticsearchIO.read().withConnectionConfiguration(
    ElasticsearchIO.ConnectionConfiguration.create("http://host:9200", "my-index", "my-type")
 )

 

The connection configuration also accepts optional configuration: withUsername() and withPassword().

You can also specify a query on the read() using withQuery().

Writing to Elasticsearch

To write documents to Elasticsearch, use ElasticsearchIO.write(), which writes JSON documents from a PCollection<String> (which can be bounded or unbounded).

To configure ElasticsearchIO.write(), similar to the read, you have to provide a connection configuration. For instance:

  pipeline
    .apply(...)
    .apply(ElasticsearchIO.write().withConnectionConfiguration(
       ElasticsearchIO.ConnectionConfiguration.create("http://host:9200", "my-index", "my-type")
    )

 

Optionally, you can provide withBatchSize() and withBatchSizeBytes() to specify the size of the write batch in number of documents or in bytes.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ElasticsearchIO.ConnectionConfiguration A POJO describing a connection configuration to Elasticsearch.
static class	ElasticsearchIO.Read A PTransform reading data from Elasticsearch.
static class	ElasticsearchIO.Write A PTransform writing data to Elasticsearch.

Method Summary

Modifier and Type	Method and Description
static ElasticsearchIO.Read	read()
static ElasticsearchIO.Write	write()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

read

public static ElasticsearchIO.Read read()

write

public static ElasticsearchIO.Write write()