@Experimental(value=SOURCE_SINK) public class ElasticsearchIO extends java.lang.Object
ElasticsearchIO.read()
returns a bounded PCollection<String>
representing JSON documents.
To configure the read()
, you have to provide a connection configuration
containing the HTTP address of the instances, an index name and a type. The following example
illustrates options for configuring the source:
pipeline.apply(ElasticsearchIO.read().withConnectionConfiguration(
ElasticsearchIO.ConnectionConfiguration.create("http://host:9200", "my-index", "my-type")
)
The connection configuration also accepts optional configuration: withUsername()
,
withPassword()
, withApiKey()
and withBearerToken()
.
You can also specify a query on the read()
using withQuery()
.
There are many more configuration options which can be found by looking at the with* methods
of ElasticsearchIO.Read
To write documents to Elasticsearch, use ElasticsearchIO.write()
, which writes JSON documents from a PCollection<String>
(which can be bounded or unbounded).
ElasticsearchIO.Write
involves 2 discrete steps:
ElasticsearchIO.DocToBulk
)
ElasticsearchIO.BulkIO
)
In most cases, using write()
will be desirable. In some cases, one may
want to use ElasticsearchIO.DocToBulk
and ElasticsearchIO.BulkIO
directly. Such
cases might include:
docToBulk()
Note that configurations options for ElasticsearchIO.Write
are a union of
configutation options for ElasticsearchIO.DocToBulk
and ElasticsearchIO.BulkIO
.
To configure ElasticsearchIO.write()
, similar to the read, you
have to provide a connection configuration. For instance:
pipeline
.apply(...)
.apply(ElasticsearchIO.write().withConnectionConfiguration(
ElasticsearchIO.ConnectionConfiguration.create("http://host:9200", "my-index", "my-type")
)
There are many more configuration options which can be found by looking at the with* methods
of ElasticsearchIO.Write
Modifier and Type | Class and Description |
---|---|
static class |
ElasticsearchIO.BoundedElasticsearchSource
A
BoundedSource reading from Elasticsearch. |
static class |
ElasticsearchIO.BulkIO
A
PTransform writing Bulk API entities created by ElasticsearchIO.DocToBulk to
an Elasticsearch cluster. |
static class |
ElasticsearchIO.ConnectionConfiguration
A POJO describing a connection configuration to Elasticsearch.
|
static class |
ElasticsearchIO.DocToBulk
A
PTransform converting docs to their Bulk API counterparts. |
static class |
ElasticsearchIO.Read
A
PTransform reading data from Elasticsearch. |
static class |
ElasticsearchIO.RetryConfiguration
A POJO encapsulating a configuration for retry behavior when issuing requests to ES.
|
static class |
ElasticsearchIO.Write
A
PTransform writing data to Elasticsearch. |
Modifier and Type | Method and Description |
---|---|
static ElasticsearchIO.BulkIO |
bulkIO() |
static ElasticsearchIO.DocToBulk |
docToBulk() |
static ElasticsearchIO.Read |
read() |
static ElasticsearchIO.Write |
write() |
public static ElasticsearchIO.Read read()
public static ElasticsearchIO.DocToBulk docToBulk()
public static ElasticsearchIO.BulkIO bulkIO()
public static ElasticsearchIO.Write write()