Package org.apache.beam.sdk.io.elasticsearch

Class ElasticsearchIO.Write

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<String>,PCollectionTuple>

org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

ElasticsearchIO

public static class ElasticsearchIO.Write
extends PTransform<PCollection<String>,PCollectionTuple>

A PTransform writing data to Elasticsearch.

This PTransform acts as a convenience wrapper for doing both document to bulk API serialization as well as batching those Bulk API entities and writing them to an Elasticsearch cluster. This class is effectively a thin proxy for DocToBulk->BulkIO all-in-one for convenience and backward compatibility.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	ElasticsearchIO.Write.BooleanFieldValueExtractFn
static interface	ElasticsearchIO.Write.FieldValueExtractFn

Field Summary

Fields

Modifier and Type	Field	Description
static final TupleTag<ElasticsearchIO.Document>	FAILED_WRITES
static final TupleTag<ElasticsearchIO.Document>	SUCCESSFUL_WRITES

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor	Description
Write()

Method Summary

Modifier and Type	Method	Description
PCollectionTuple	expand(PCollection<String> input)	Override this method to specify how this PTransform should be expanded on the given InputT.
ElasticsearchIO.BulkIO	getBulkIO()
ElasticsearchIO.DocToBulk	getDocToBulk()
ElasticsearchIO.Write	withAllowableResponseErrors(@Nullable Set<String> allowableResponseErrors)	Refer to ElasticsearchIO.BulkIO.withAllowableResponseErrors(java.util.Set<java.lang.String>).
ElasticsearchIO.Write	withAppendOnly(boolean appendOnly)	Refer to ElasticsearchIO.DocToBulk.withAppendOnly(boolean).
ElasticsearchIO.Write	withBackendVersion(int backendVersion)	Refer to ElasticsearchIO.DocToBulk.withBackendVersion(int).
ElasticsearchIO.Write	withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration)	Refer to ElasticsearchIO.BulkIO.withConnectionConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration).
ElasticsearchIO.Write	withDocVersionFn(ElasticsearchIO.Write.FieldValueExtractFn docVersionFn)	Refer to ElasticsearchIO.DocToBulk.withDocVersionFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withDocVersionType(String docVersionType)	Refer to ElasticsearchIO.DocToBulk.withDocVersionType(java.lang.String).
ElasticsearchIO.Write	withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn)	Refer to ElasticsearchIO.DocToBulk.withIdFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withIgnoreVersionConflicts(boolean ignoreVersionConflicts)	Refer to ElasticsearchIO.BulkIO.withIgnoreVersionConflicts(boolean).
ElasticsearchIO.Write	withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn)	Refer to ElasticsearchIO.DocToBulk.withIndexFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withIsDeleteFn(ElasticsearchIO.Write.BooleanFieldValueExtractFn isDeleteFn)	Refer to ElasticsearchIO.DocToBulk.withIsDeleteFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.BooleanFieldValueExtractFn).
ElasticsearchIO.Write	withMaxBatchSize(long batchSize)	Refer to ElasticsearchIO.BulkIO.withMaxBatchSize(long).
ElasticsearchIO.Write	withMaxBatchSizeBytes(long batchSizeBytes)	Refer to ElasticsearchIO.BulkIO.withMaxBatchSizeBytes(long).
ElasticsearchIO.Write	withMaxBufferingDuration(Duration maxBufferingDuration)	Refer to ElasticsearchIO.BulkIO.withMaxBufferingDuration(org.joda.time.Duration).
ElasticsearchIO.Write	withMaxParallelRequests(int maxParallelRequests)	Refer to ElasticsearchIO.BulkIO.withMaxParallelRequests(int).
ElasticsearchIO.Write	withMaxParallelRequestsPerWindow(int maxParallelRequestsPerWindow)	Deprecated. use withMaxParallelRequests(int) instead
ElasticsearchIO.Write	withRetryConfiguration(ElasticsearchIO.RetryConfiguration retryConfiguration)	Refer to ElasticsearchIO.BulkIO.withRetryConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration).
ElasticsearchIO.Write	withRoutingFn(ElasticsearchIO.Write.FieldValueExtractFn routingFn)	Refer to ElasticsearchIO.DocToBulk.withRoutingFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withThrowWriteErrors(boolean throwWriteErrors)	Refer to ElasticsearchIO.BulkIO.withThrowWriteErrors(boolean).
ElasticsearchIO.Write	withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn)	Refer to ElasticsearchIO.DocToBulk.withTypeFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).
ElasticsearchIO.Write	withUpsertScript(String source)	Refer to ElasticsearchIO.DocToBulk.withUpsertScript(java.lang.String).
ElasticsearchIO.Write	withUsePartialUpdate(boolean usePartialUpdate)	Refer to ElasticsearchIO.DocToBulk.withUsePartialUpdate(boolean).
ElasticsearchIO.Write	withUseStatefulBatches(boolean useStatefulBatches)	Refer to ElasticsearchIO.BulkIO.withUseStatefulBatches(boolean).

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

SUCCESSFUL_WRITES

public static final TupleTag<ElasticsearchIO.Document> SUCCESSFUL_WRITES

FAILED_WRITES

public static final TupleTag<ElasticsearchIO.Document> FAILED_WRITES

Constructor Details

Write

public Write()

Method Details

getDocToBulk

public ElasticsearchIO.DocToBulk getDocToBulk()

getBulkIO

public ElasticsearchIO.BulkIO getBulkIO()

withIdFn

public ElasticsearchIO.Write withIdFn(ElasticsearchIO.Write.FieldValueExtractFn idFn)

Refer to ElasticsearchIO.DocToBulk.withIdFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withIndexFn

public ElasticsearchIO.Write withIndexFn(ElasticsearchIO.Write.FieldValueExtractFn indexFn)

Refer to ElasticsearchIO.DocToBulk.withIndexFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withRoutingFn

public ElasticsearchIO.Write withRoutingFn(ElasticsearchIO.Write.FieldValueExtractFn routingFn)

Refer to ElasticsearchIO.DocToBulk.withRoutingFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withTypeFn

public ElasticsearchIO.Write withTypeFn(ElasticsearchIO.Write.FieldValueExtractFn typeFn)

Refer to ElasticsearchIO.DocToBulk.withTypeFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withDocVersionFn

public ElasticsearchIO.Write withDocVersionFn(ElasticsearchIO.Write.FieldValueExtractFn docVersionFn)

Refer to ElasticsearchIO.DocToBulk.withDocVersionFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.FieldValueExtractFn).

withDocVersionType

public ElasticsearchIO.Write withDocVersionType(String docVersionType)

Refer to ElasticsearchIO.DocToBulk.withDocVersionType(java.lang.String).

withUsePartialUpdate

public ElasticsearchIO.Write withUsePartialUpdate(boolean usePartialUpdate)

Refer to ElasticsearchIO.DocToBulk.withUsePartialUpdate(boolean).

withAppendOnly

public ElasticsearchIO.Write withAppendOnly(boolean appendOnly)

Refer to ElasticsearchIO.DocToBulk.withAppendOnly(boolean).

withUpsertScript

public ElasticsearchIO.Write withUpsertScript(String source)

Refer to ElasticsearchIO.DocToBulk.withUpsertScript(java.lang.String).

withBackendVersion

public ElasticsearchIO.Write withBackendVersion(int backendVersion)

Refer to ElasticsearchIO.DocToBulk.withBackendVersion(int).

withIsDeleteFn

public ElasticsearchIO.Write withIsDeleteFn(ElasticsearchIO.Write.BooleanFieldValueExtractFn isDeleteFn)

Refer to ElasticsearchIO.DocToBulk.withIsDeleteFn(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write.BooleanFieldValueExtractFn).

withConnectionConfiguration

public ElasticsearchIO.Write withConnectionConfiguration(ElasticsearchIO.ConnectionConfiguration connectionConfiguration)

Refer to ElasticsearchIO.BulkIO.withConnectionConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration).

withMaxBatchSize

public ElasticsearchIO.Write withMaxBatchSize(long batchSize)

Refer to ElasticsearchIO.BulkIO.withMaxBatchSize(long).

withMaxBatchSizeBytes

public ElasticsearchIO.Write withMaxBatchSizeBytes(long batchSizeBytes)

Refer to ElasticsearchIO.BulkIO.withMaxBatchSizeBytes(long).

withRetryConfiguration

public ElasticsearchIO.Write withRetryConfiguration(ElasticsearchIO.RetryConfiguration retryConfiguration)

Refer to ElasticsearchIO.BulkIO.withRetryConfiguration(org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration).

withIgnoreVersionConflicts

public ElasticsearchIO.Write withIgnoreVersionConflicts(boolean ignoreVersionConflicts)

Refer to ElasticsearchIO.BulkIO.withIgnoreVersionConflicts(boolean).

withUseStatefulBatches

public ElasticsearchIO.Write withUseStatefulBatches(boolean useStatefulBatches)

Refer to ElasticsearchIO.BulkIO.withUseStatefulBatches(boolean).

withMaxBufferingDuration

public ElasticsearchIO.Write withMaxBufferingDuration(Duration maxBufferingDuration)

Refer to ElasticsearchIO.BulkIO.withMaxBufferingDuration(org.joda.time.Duration).

withMaxParallelRequestsPerWindow

@Deprecated
public ElasticsearchIO.Write withMaxParallelRequestsPerWindow(int maxParallelRequestsPerWindow)

Deprecated.

use withMaxParallelRequests(int) instead

Refer to ElasticsearchIO.BulkIO.withMaxParallelRequestsPerWindow(int).

withMaxParallelRequests

public ElasticsearchIO.Write withMaxParallelRequests(int maxParallelRequests)

Refer to ElasticsearchIO.BulkIO.withMaxParallelRequests(int).

withAllowableResponseErrors

public ElasticsearchIO.Write withAllowableResponseErrors(@Nullable Set<String> allowableResponseErrors)

Refer to ElasticsearchIO.BulkIO.withAllowableResponseErrors(java.util.Set<java.lang.String>).

withThrowWriteErrors

public ElasticsearchIO.Write withThrowWriteErrors(boolean throwWriteErrors)

Refer to ElasticsearchIO.BulkIO.withThrowWriteErrors(boolean).

expand

public PCollectionTuple expand(PCollection<String> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<String>,PCollectionTuple>