apache_beam.utils.profiler module

A profiler context manager based on cProfile.Profile objects.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.utils.profiler.Profile(profile_id, profile_location=None, log_results=False, file_copy_fn=None, time_prefix='%Y-%m-%d_%H_%M_%S-')

Bases: object

cProfile wrapper context for saving and logging profiler results.

SORTBY = 'cumulative'

static default_file_copy_fn(src, dest)

static factory_from_options(options)

class apache_beam.utils.profiler.MemoryReporter(interval_second=60.0)

Bases: object

A memory reporter that reports the memory usage and heap profile. Usage::

mr = MemoryReporter(interval_second=30.0)
mr.start()
while ...
  <do something>
  # this will report continuously with 30 seconds between reports.
mr.stop()

NOTE: A reporter with start() should always stop(), or the parent process can never finish.

Or simply the following which does star() and stop():

with MemoryReporter(interval_second=100):

while …

<do some thing>

Also it could report on demand without continuous reporting.:

mr = MemoryReporter()  # default interval 60s but not started.
<do something>
mr.report_once()

start()

stop()

report_once()