apache_beam.testing.benchmarks.nexmark.nexmark_launcher module

Nexmark launcher.

The Nexmark suite is a series of queries (streaming pipelines) performed on a simulation of auction events. The launcher orchestrates the generation and parsing of streaming events and the running of queries.

Model
  • Person: Author of an auction or a bid.
  • Auction: Item under auction.
  • Bid: A bid for an item under auction.
Events
  • Create Person
  • Create Auction
  • Create Bid
Queries
  • Query0: Pass through (send and receive auction events).
Usage
  • DirectRunner
    python nexmark_launcher.py –query/q <query number> –project <project id> –loglevel=DEBUG (optional) –wait_until_finish_duration <time_in_ms> –streaming
  • DataflowRunner
    python nexmark_launcher.py –query/q <query number> –project <project id> –region <GCE region> –loglevel=DEBUG (optional) –wait_until_finish_duration <time_in_ms> –streaming –sdk_location <apache_beam tar.gz> –staging_location=gs://… –temp_location=gs://
class apache_beam.testing.benchmarks.nexmark.nexmark_launcher.NexmarkLauncher[source]

Bases: object

DONE_DELAY = 300
PERF_DELAY = 20
TERMINATE_DELAY = 3600
WARNING_DELAY = 600
parse_args()[source]
generate_events()[source]
read_from_file()[source]
read_from_pubsub()[source]
run_query(query_num, query, query_args, pipeline_options, query_errors)[source]
monitor(job, event_monitor, result_monitor)[source]

keep monitoring the performance and progress of running job and cancel the job if the job is stuck or seems to have finished running

Returns:the final performance if it is measured
static log_performance(perf: apache_beam.testing.benchmarks.nexmark.nexmark_perf.NexmarkPerf) → None[source]
publish_performance_influxdb(query_num, perf)[source]
static get_performance(result, event_monitor, result_monitor)[source]
cleanup()[source]
run()[source]