Package org.apache.beam.runners.spark.metrics.sink

Class CsvSink

java.lang.Object
org.apache.beam.runners.spark.metrics.sink.CsvSink
All Implemented Interfaces:
org.apache.spark.metrics.sink.Sink
public class CsvSink extends Object implements org.apache.spark.metrics.sink.Sink
A Sink for Spark's metric system reporting metrics (including Beam step metrics) to a CSV file.

The sink is configured using Spark configuration parameters, for example: 


 "spark.metrics.conf.*.sink.csv.class"="org.apache.beam.runners.spark.metrics.sink.CsvSink"
 "spark.metrics.conf.*.sink.csv.directory"="<output_directory>"
 "spark.metrics.conf.*.sink.csv.period"=10
 "spark.metrics.conf.*.sink.csv.unit"=seconds

  • Constructor Summary

    Constructors
    Constructor
    Description
    CsvSink(Properties properties, com.codahale.metrics.MetricRegistry metricRegistry)
    Constructor for Spark 3.2.x and later.
    CsvSink(Properties properties, com.codahale.metrics.MetricRegistry metricRegistry, org.apache.spark.SecurityManager securityMgr)
    Constructor for Spark 3.1.x and earlier.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    report()
     
    void
    start()
     
    void
    stop()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • CsvSink

      public CsvSink(Properties properties, com.codahale.metrics.MetricRegistry metricRegistry, org.apache.spark.SecurityManager securityMgr)
      Constructor for Spark 3.1.x and earlier.

    • CsvSink

      public CsvSink(Properties properties, com.codahale.metrics.MetricRegistry metricRegistry)
      Constructor for Spark 3.2.x and later.

  • Method Details

    • start

      public void start()
      Specified by:
      start in interface org.apache.spark.metrics.sink.Sink

    • stop

      public void stop()
      Specified by:
      stop in interface org.apache.spark.metrics.sink.Sink

    • report

      public void report()
      Specified by:
      report in interface org.apache.spark.metrics.sink.Sink