Package org.apache.beam.sdk.io.splunk

Class SplunkIO

java.lang.Object
org.apache.beam.sdk.io.splunk.SplunkIO
public class SplunkIO extends Object
An unbounded sink for Splunk's Http Event Collector (HEC).

For more information, see the online documentation at Splunk HEC.

Writing to Splunk's HEC

The SplunkIO class provides a PTransform that allows writing SplunkEvent messages into a Splunk HEC end point.

It takes as an input a PCollection<SplunkEvent>, where each SplunkEvent represents an event to be published to HEC.

To configure a SplunkIO, you must provide at a minimum:

  • url - HEC endpoint URL.
  • token - HEC endpoint token.

The SplunkIO transform can be customized further by optionally specifying:

  • parallelism - Number of parallel requests to the HEC.
  • batchCount - Number of events in a single batch.
  • disableCertificateValidation - Whether to disable ssl validation (useful for self-signed certificates)
  • enableBatchLogs - Whether to enable batch logs.
  • enableGzipHttpCompression - Whether HTTP requests sent to Splunk HEC should be GZIP encoded.

This transform will return any non-transient write failures via a PCollection<SplunkWriteError>, where each SplunkWriteError captures the error that occurred while attempting to write to HEC. These can be published to a dead-letter sink or reprocessed.

For example: 


 PCollection<SplunkEvent> events = ...;

 PCollection<SplunkWriteError> errors =
         events.apply("WriteToSplunk",
              SplunkIO.write(url, token)
                  .withBatchCount(batchCount)
                  .withParallelism(parallelism)
                  .withDisableCertificateValidation(true));