org.apache.beam.sdk.io.elasticsearch

Class ElasticsearchIO.Write

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<java.lang.String>,PDone>

org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

ElasticsearchIO

public abstract static class ElasticsearchIO.Write
extends PTransform<PCollection<java.lang.String>,PDone>

A PTransform writing data to Elasticsearch.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	ElasticsearchIO.Write.FieldValueExtractFn Interface allowing a specific field value to be returned from a parsed JSON document.

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<java.lang.String> input) Override this method to specify how this PTransform should be expanded on the given InputT.
ElasticsearchIO.Write	withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration) Provide the Elasticsearch connection configuration object.
ElasticsearchIO.Write	withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn) Provide a function to extract the id from the document.
ElasticsearchIO.Write	withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn) Provide a function to extract the target index from the document allowing for dynamic document routing.
ElasticsearchIO.Write	withMaxBatchSize(long batchSize) Provide a maximum size in number of documents for the batch see bulk API (https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-bulk.html).
ElasticsearchIO.Write	withMaxBatchSizeBytes(long batchSizeBytes) Provide a maximum size in bytes for the batch see bulk API (https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-bulk.html).
ElasticsearchIO.Write	withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn) Provide a function to extract the target type from the document allowing for dynamic document routing.
ElasticsearchIO.Write	withUsePartialUpdate(boolean usePartialUpdate) Provide an instruction to control whether partial updates or inserts (default) are issued to Elasticsearch.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

withConnectionConfiguration

public ElasticsearchIO.Write withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration)

Provide the Elasticsearch connection configuration object.

Parameters:

connectionConfiguration - the Elasticsearch ElasticsearchIO.ConnectionConfiguration object

Returns:

the ElasticsearchIO.Write with connection configuration set

withMaxBatchSize

public ElasticsearchIO.Write withMaxBatchSize(long batchSize)

Provide a maximum size in number of documents for the batch see bulk API (https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-bulk.html). Default is 1000 docs (like Elasticsearch bulk size advice). See https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html Depending on the execution engine, size of bundles may vary, this sets the maximum size. Change this if you need to have smaller ElasticSearch bulks.

Parameters:

batchSize - maximum batch size in number of documents

Returns:

the ElasticsearchIO.Write with connection batch size set

withMaxBatchSizeBytes

public ElasticsearchIO.Write withMaxBatchSizeBytes(long batchSizeBytes)

Provide a maximum size in bytes for the batch see bulk API (https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-bulk.html). Default is 5MB (like Elasticsearch bulk size advice). See https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html Depending on the execution engine, size of bundles may vary, this sets the maximum size. Change this if you need to have smaller ElasticSearch bulks.

Parameters:

batchSizeBytes - maximum batch size in bytes

Returns:

the ElasticsearchIO.Write with connection batch size in bytes set

withIdFn

public ElasticsearchIO.Write withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn)

Provide a function to extract the id from the document. This id will be used as the document id in Elasticsearch. Should the function throw an Exception then the batch will fail and the exception propagated.

Parameters:

idFn - to extract the document ID

Returns:

the ElasticsearchIO.Write with the function set

withIndexFn

public ElasticsearchIO.Write withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn)

Provide a function to extract the target index from the document allowing for dynamic document routing. Should the function throw an Exception then the batch will fail and the exception propagated.

Parameters:

indexFn - to extract the destination index from

Returns:

the ElasticsearchIO.Write with the function set

withTypeFn

public ElasticsearchIO.Write withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn)

Provide a function to extract the target type from the document allowing for dynamic document routing. Should the function throw an Exception then the batch will fail and the exception propagated. Users are encouraged to consider carefully if multipe types are a sensible model as discussed in this blog.

Parameters:

typeFn - to extract the destination index from

Returns:

the ElasticsearchIO.Write with the function set

withUsePartialUpdate

public ElasticsearchIO.Write withUsePartialUpdate(boolean usePartialUpdate)

Provide an instruction to control whether partial updates or inserts (default) are issued to Elasticsearch.

Parameters:

usePartialUpdate - set to true to issue partial updates

Returns:

the ElasticsearchIO.Write with the partial update control set

expand

public PDone expand(PCollection<java.lang.String> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<java.lang.String>,PDone>