org.apache.beam.sdk.io.solr

Class SolrIO.Write

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<org.apache.solr.common.SolrInputDocument>,PDone>

org.apache.beam.sdk.io.solr.SolrIO.Write

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Enclosing class:

SolrIO

public abstract static class SolrIO.Write
extends PTransform<PCollection<org.apache.solr.common.SolrInputDocument>,PDone>

A PTransform writing data to Solr.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<org.apache.solr.common.SolrInputDocument> input) Override this method to specify how this PTransform should be expanded on the given InputT.
SolrIO.Write	to(java.lang.String collection) Provide name of collection while reading from Solr.
SolrIO.Write	withConnectionConfiguration(SolrIO.ConnectionConfiguration connectionConfiguration) Provide the Solr connection configuration object.
SolrIO.Write	withMaxBatchSize(int batchSize) Provide a maximum size in number of documents for the batch.
SolrIO.Write	withRetryConfiguration(SolrIO.RetryConfiguration retryConfiguration) Provides configuration to retry a failed batch call to Solr.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

withConnectionConfiguration

public SolrIO.Write withConnectionConfiguration(SolrIO.ConnectionConfiguration connectionConfiguration)

Provide the Solr connection configuration object.

to

public SolrIO.Write to(java.lang.String collection)

Provide name of collection while reading from Solr.

Parameters:

collection - the collection toward which the requests will be issued

withMaxBatchSize

public SolrIO.Write withMaxBatchSize(int batchSize)

Provide a maximum size in number of documents for the batch. Depending on the execution engine, size of bundles may vary, this sets the maximum size. Change this if you need to have smaller batch. Default max batch size is 1000.

Parameters:

batchSize - maximum batch size in number of documents

withRetryConfiguration

public SolrIO.Write withRetryConfiguration(SolrIO.RetryConfiguration retryConfiguration)

Provides configuration to retry a failed batch call to Solr. A batch is considered as failed if the underlying CloudSolrClient surfaces HttpSolrClient.RemoteSolrException, SolrServerException or IOException. Users should consider that retrying might compound the underlying problem which caused the initial failure. Users should also be aware that once retrying is exhausted the error is surfaced to the runner which may then opt to retry the current partition in entirety or abort if the max number of retries of the runner is completed. Retrying uses an exponential backoff algorithm, with minimum backoff of 5 seconds and then surfacing the error once the maximum number of retries or maximum configuration duration is exceeded.

Example use:

 SolrIO.write()
   .withRetryConfiguration(SolrIO.RetryConfiguration.create(10, Duration.standardMinutes(3))
   ...
 

Parameters:

retryConfiguration - the rules which govern the retry behavior

Returns:

the SolrIO.Write with retrying configured

expand

public PDone expand(PCollection<org.apache.solr.common.SolrInputDocument> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<org.apache.solr.common.SolrInputDocument>,PDone>