public abstract static class ElasticsearchIO.Write extends PTransform<PCollection<java.lang.String>,PDone>
PTransform writing data to Elasticsearch.| Modifier and Type | Class and Description |
|---|---|
static interface |
ElasticsearchIO.Write.BooleanFieldValueExtractFn |
static interface |
ElasticsearchIO.Write.FieldValueExtractFn
Interface allowing a specific field value to be returned from a parsed JSON document.
|
name, resourceHints| Constructor and Description |
|---|
Write() |
| Modifier and Type | Method and Description |
|---|---|
PDone |
expand(PCollection<java.lang.String> input)
Override this method to specify how this
PTransform should be expanded on the given
InputT. |
ElasticsearchIO.Write |
withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration)
Provide the Elasticsearch connection configuration object.
|
ElasticsearchIO.Write |
withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn)
Provide a function to extract the id from the document.
|
ElasticsearchIO.Write |
withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn)
Provide a function to extract the target index from the document allowing for dynamic
document routing.
|
ElasticsearchIO.Write |
withIsDeleteFn(ElasticsearchIO.Write.BooleanFieldValueExtractFn isDeleteFn)
Provide a function to extract the target operation either upsert or delete from the document
fields allowing dynamic bulk operation decision.
|
ElasticsearchIO.Write |
withMaxBatchSize(long batchSize)
Provide a maximum size in number of documents for the batch see bulk API
(https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-bulk.html).
|
ElasticsearchIO.Write |
withMaxBatchSizeBytes(long batchSizeBytes)
Provide a maximum size in bytes for the batch see bulk API
(https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-bulk.html).
|
ElasticsearchIO.Write |
withRetryConfiguration(ElasticsearchIO.RetryConfiguration retryConfiguration)
Provides configuration to retry a failed batch call to Elasticsearch.
|
ElasticsearchIO.Write |
withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn)
Provide a function to extract the target type from the document allowing for dynamic document
routing.
|
ElasticsearchIO.Write |
withUsePartialUpdate(boolean usePartialUpdate)
Provide an instruction to control whether partial updates or inserts (default) are issued to
Elasticsearch.
|
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validatepublic ElasticsearchIO.Write withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration)
connectionConfiguration - the Elasticsearch ElasticsearchIO.ConnectionConfiguration objectElasticsearchIO.Write with connection configuration setpublic ElasticsearchIO.Write withMaxBatchSize(long batchSize)
batchSize - maximum batch size in number of documentsElasticsearchIO.Write with connection batch size setpublic ElasticsearchIO.Write withMaxBatchSizeBytes(long batchSizeBytes)
batchSizeBytes - maximum batch size in bytesElasticsearchIO.Write with connection batch size in bytes setpublic ElasticsearchIO.Write withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn)
idFn - to extract the document IDElasticsearchIO.Write with the function setpublic ElasticsearchIO.Write withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn)
indexFn - to extract the destination index fromElasticsearchIO.Write with the function setpublic ElasticsearchIO.Write withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn)
typeFn - to extract the destination index fromElasticsearchIO.Write with the function setpublic ElasticsearchIO.Write withUsePartialUpdate(boolean usePartialUpdate)
usePartialUpdate - set to true to issue partial updatesElasticsearchIO.Write with the partial update control setpublic ElasticsearchIO.Write withRetryConfiguration(ElasticsearchIO.RetryConfiguration retryConfiguration)
RestClient surfaces 429 HTTP status code as error for one
or more of the items in the Response. Users should consider that retrying might
compound the underlying problem which caused the initial failure. Users should also be aware
that once retrying is exhausted the error is surfaced to the runner which may then
opt to retry the current bundle in entirety or abort if the max number of retries of the
runner is completed. Retrying uses an exponential backoff algorithm, with minimum backoff of
5 seconds and then surfacing the error once the maximum number of retries or maximum
configuration duration is exceeded.
Example use:
ElasticsearchIO.write()
.withRetryConfiguration(ElasticsearchIO.RetryConfiguration.create(10, Duration.standardMinutes(3))
...
retryConfiguration - the rules which govern the retry behaviorElasticsearchIO.Write with retrying configuredpublic ElasticsearchIO.Write withIsDeleteFn(ElasticsearchIO.Write.BooleanFieldValueExtractFn isDeleteFn)
isDeleteFn - set to true for deleting the specific documentElasticsearchIO.Write with the function setpublic PDone expand(PCollection<java.lang.String> input)
PTransformPTransform should be expanded on the given
InputT.
NOTE: This method should not be called directly. Instead apply the PTransform should
be applied to the InputT using the apply method.
Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
expand in class PTransform<PCollection<java.lang.String>,PDone>