apache_beam.utils.profiler module

A profiler context manager based on cProfile.Profile objects.

For internal use only; no backwards-compatibility guarantees.

class apache_beam.utils.profiler.Profile(profile_id, profile_location=None, log_results=False, file_copy_fn=None, time_prefix='%Y-%m-%d_%H_%M_%S-')[source]

Bases: future.types.newobject.newobject

cProfile wrapper context for saving and logging profiler results.

SORTBY = 'cumulative'
static default_file_copy_fn(src, dest)[source]
static factory_from_options(options)[source]
class apache_beam.utils.profiler.MemoryReporter(interval_second=60.0)[source]

Bases: future.types.newobject.newobject

A memory reporter that reports the memory usage and heap profile. Usage::

mr = MemoryReporter(interval_second=30.0)
mr.start()
while ...
  <do something>
  # this will report continuously with 30 seconds between reports.
mr.stop()

NOTE: A reporter with start() should always stop(), or the parent process can never finish.

Or simply the following which does star() and stop():
with MemoryReporter(interval_second=100):
while …
<do some thing>

Also it could report on demand without continuous reporting.:

mr = MemoryReporter()  # default interval 60s but not started.
<do something>
mr.report_once()
start()[source]
stop()[source]
report_once()[source]