org.apache.beam.sdk.io.elasticsearch

Class ElasticsearchIO.DocToBulk

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<java.lang.String>,PCollection<java.lang.String>>

org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.DocToBulk

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

ElasticsearchIO

public abstract static class ElasticsearchIO.DocToBulk
extends PTransform<PCollection<java.lang.String>,PCollection<java.lang.String>>

A PTransform converting docs to their Bulk API counterparts.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
DocToBulk()

Method Summary

Modifier and Type	Method and Description
PCollection<java.lang.String>	expand(PCollection<java.lang.String> docs) Override this method to specify how this PTransform should be expanded on the given InputT.
ElasticsearchIO.DocToBulk	withBackendVersion(int backendVersion) Use to set explicitly which version of Elasticsearch the destination cluster is running.
ElasticsearchIO.DocToBulk	withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration) Provide the Elasticsearch connection configuration object.
ElasticsearchIO.DocToBulk	withDocVersionFn(ElasticsearchIO.Write.FieldValueExtractFn docVersionFn) Provide a function to extract the doc version from the document.
ElasticsearchIO.DocToBulk	withDocVersionType(java.lang.String docVersionType) Provide a function to extract the doc version from the document.
ElasticsearchIO.DocToBulk	withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn) Provide a function to extract the id from the document.
ElasticsearchIO.DocToBulk	withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn) Provide a function to extract the target index from the document allowing for dynamic document routing.
ElasticsearchIO.DocToBulk	withIsDeleteFn(ElasticsearchIO.Write.BooleanFieldValueExtractFn isDeleteFn) Provide a function to extract the target operation either upsert or delete from the document fields allowing dynamic bulk operation decision.
ElasticsearchIO.DocToBulk	withRoutingFn(ElasticsearchIO.Write.FieldValueExtractFn routingFn) Provide a function to extract the target routing from the document allowing for dynamic document routing.
ElasticsearchIO.DocToBulk	withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn) Provide a function to extract the target type from the document allowing for dynamic document routing.
ElasticsearchIO.DocToBulk	withUpsertScript(java.lang.String source) Whether to use scripted updates and what script to use.
ElasticsearchIO.DocToBulk	withUsePartialUpdate(boolean usePartialUpdate) Provide an instruction to control whether partial updates or inserts (default) are issued to Elasticsearch.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DocToBulk

public DocToBulk()

Method Detail

withConnectionConfiguration

public ElasticsearchIO.DocToBulk withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration)

Provide the Elasticsearch connection configuration object. Only required if withBackendVersion was not used i.e. getBackendVersion() returns null.

Parameters:

connectionConfiguration - the Elasticsearch ElasticsearchIO.ConnectionConfiguration object

Returns:

the ElasticsearchIO.DocToBulk with connection configuration set

withIdFn

public ElasticsearchIO.DocToBulk withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn)

Provide a function to extract the id from the document. This id will be used as the document id in Elasticsearch. Should the function throw an Exception then the batch will fail and the exception propagated.

Parameters:

idFn - to extract the document ID

Returns:

the ElasticsearchIO.DocToBulk with the function set

withIndexFn

public ElasticsearchIO.DocToBulk withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn)

Provide a function to extract the target index from the document allowing for dynamic document routing. Should the function throw an Exception then the batch will fail and the exception propagated.

Parameters:

indexFn - to extract the destination index from

Returns:

the ElasticsearchIO.DocToBulk with the function set

withRoutingFn

public ElasticsearchIO.DocToBulk withRoutingFn(ElasticsearchIO.Write.FieldValueExtractFn routingFn)

Provide a function to extract the target routing from the document allowing for dynamic document routing. Should the function throw an Exception then the batch will fail and the exception propagated.

Parameters:

routingFn - to extract the destination index from

Returns:

the ElasticsearchIO.DocToBulk with the function set

withTypeFn

public ElasticsearchIO.DocToBulk withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn)

Provide a function to extract the target type from the document allowing for dynamic document routing. Should the function throw an Exception then the batch will fail and the exception propagated. Users are encouraged to consider carefully if multipe types are a sensible model as discussed in this blog.

Parameters:

typeFn - to extract the destination index from

Returns:

the ElasticsearchIO.DocToBulk with the function set

withUsePartialUpdate

public ElasticsearchIO.DocToBulk withUsePartialUpdate(boolean usePartialUpdate)

Provide an instruction to control whether partial updates or inserts (default) are issued to Elasticsearch.

Parameters:

usePartialUpdate - set to true to issue partial updates

Returns:

the ElasticsearchIO.DocToBulk with the partial update control set

withUpsertScript

public ElasticsearchIO.DocToBulk withUpsertScript(java.lang.String source)

Whether to use scripted updates and what script to use.

Parameters:

source - set to the value of the script source, painless lang

Returns:

the ElasticsearchIO.DocToBulk with the scripted updates set

withDocVersionFn

public ElasticsearchIO.DocToBulk withDocVersionFn(ElasticsearchIO.Write.FieldValueExtractFn docVersionFn)

Provide a function to extract the doc version from the document. This version number will be used as the document version in Elasticsearch. Should the function throw an Exception then the batch will fail and the exception propagated. Incompatible with update operations and should only be used with withUsePartialUpdate(false)

Parameters:

docVersionFn - to extract the document version

Returns:

the ElasticsearchIO.DocToBulk with the function set

withIsDeleteFn

public ElasticsearchIO.DocToBulk withIsDeleteFn(ElasticsearchIO.Write.BooleanFieldValueExtractFn isDeleteFn)

Provide a function to extract the target operation either upsert or delete from the document fields allowing dynamic bulk operation decision. While using withIsDeleteFn, it should be taken care that the document's id extraction is defined using the withIdFn function or else IllegalArgumentException is thrown. Should the function throw an Exception then the batch will fail and the exception propagated.

Parameters:

isDeleteFn - set to true for deleting the specific document

Returns:

the ElasticsearchIO.Write with the function set

withDocVersionType

public ElasticsearchIO.DocToBulk withDocVersionType(java.lang.String docVersionType)

Provide a function to extract the doc version from the document. This version number will be used as the document version in Elasticsearch. Should the function throw an Exception then the batch will fail and the exception propagated. Incompatible with update operations and should only be used with withUsePartialUpdate(false)

Parameters:

docVersionType - the version type to use, one of

Returns:

the ElasticsearchIO.DocToBulk with the doc version type set

withBackendVersion

public ElasticsearchIO.DocToBulk withBackendVersion(int backendVersion)

Use to set explicitly which version of Elasticsearch the destination cluster is running. Providing this hint means there is no need for setting withConnectionConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration). This can also be very useful for testing purposes.

Note: if the value of @param backendVersion differs from the version the destination cluster is running, behavior is undefined and likely to yield errors.

Parameters:

backendVersion - the major version number of the version of Elasticsearch being run in the cluster where documents will be indexed.

Returns:

the ElasticsearchIO.DocToBulk with the Elasticsearch major version number set

expand

public PCollection<java.lang.String> expand(PCollection<java.lang.String> docs)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<java.lang.String>,PCollection<java.lang.String>>