Package org.apache.beam.sdk.io.solr

Class SolrIO.Write

java.lang.Object
org.apache.beam.sdk.transforms.PTransform<PCollection<org.apache.solr.common.SolrInputDocument>,PDone>
org.apache.beam.sdk.io.solr.SolrIO.Write
All Implemented Interfaces:
Serializable, HasDisplayData
Enclosing class:
SolrIO
public abstract static class SolrIO.Write extends PTransform<PCollection<org.apache.solr.common.SolrInputDocument>,PDone>
A PTransform writing data to Solr.
See Also:

  • Constructor Details

    • Write

      public Write()

  • Method Details

    • withConnectionConfiguration

      public SolrIO.Write withConnectionConfiguration(SolrIO.ConnectionConfiguration connectionConfiguration)
      Provide the Solr connection configuration object.

    • to

      public SolrIO.Write to(String collection)
      Provide name of collection while reading from Solr.
      Parameters:
      collection - the collection toward which the requests will be issued

    • withMaxBatchSize

      public SolrIO.Write withMaxBatchSize(int batchSize)
      Provide a maximum size in number of documents for the batch. Depending on the execution engine, size of bundles may vary, this sets the maximum size. Change this if you need to have smaller batch. Default max batch size is 1000.
      Parameters:
      batchSize - maximum batch size in number of documents

    • withRetryConfiguration

      public SolrIO.Write withRetryConfiguration(SolrIO.RetryConfiguration retryConfiguration)
      Provides configuration to retry a failed batch call to Solr. A batch is considered as failed if the underlying CloudSolrClient surfaces HttpSolrClient.RemoteSolrException, SolrServerException or IOException. Users should consider that retrying might compound the underlying problem which caused the initial failure. Users should also be aware that once retrying is exhausted the error is surfaced to the runner which may then opt to retry the current partition in entirety or abort if the max number of retries of the runner is completed. Retrying uses an exponential backoff algorithm, with minimum backoff of 5 seconds and then surfacing the error once the maximum number of retries or maximum configuration duration is exceeded.

      Example use: 

      
 SolrIO.write()
   .withRetryConfiguration(SolrIO.RetryConfiguration.create(10, Duration.standardMinutes(3))
   ...
      Parameters:
      retryConfiguration - the rules which govern the retry behavior
      Returns:
      the SolrIO.Write with retrying configured

    • expand

      public PDone expand(PCollection<org.apache.solr.common.SolrInputDocument> input)
      Description copied from class: PTransform
      Override this method to specify how this PTransform should be expanded on the given InputT.

      NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      Specified by:
      expand in class PTransform<PCollection<org.apache.solr.common.SolrInputDocument>,PDone>